[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:45:45 dist-test-slave-8hhm krb5kdc[15299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:45:45 dist-test-slave-8hhm krb5kdc[15299](info): set up 2 sockets
Aug 10 04:45:45 dist-test-slave-8hhm krb5kdc[15299](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 04:45:46.627436 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:45:46 dist-test-slave-8hhm krb5kdc[15299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:45:46.748162 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:36223
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:46.832311 15317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:46.832526 15317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:46.832551 15317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:46.833755 15317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:45:46.833801 15317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:46.833815 15317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:46.833829 15317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:45:46.833878 15317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:45:46.835268 15317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:46.835507 15317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:46.835726 15317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:46.840880 15329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:46.840857 15317 server_base.cc:1034] running on GCE node
W20260810 04:45:46.840840 15327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:46.840840 15325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:46.841382 15317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:46.841682 15317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:46.842826 15317 hybrid_clock.cc:648] HybridClock initialized: now 1786337146842823 us; error 113 us; skew 500 ppm
Aug 10 04:45:46 dist-test-slave-8hhm krb5kdc[15299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337146, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:46.845432 15317 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:45:46.845885 15317 webserver.cc:460] Webserver started at http://127.14.237.62:44529/ using document root <none> and password file <none>
I20260810 04:45:46.846118 15317 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:46.846187 15317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:46.846338 15317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:46.847401 15317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "c61835810a8543f280a65e37be1bdd46"
format_stamp: "Formatted at 2026-08-10 04:45:46 on dist-test-slave-8hhm"
server_key: "5059fa3dd6f4cb311aefcc523b357f3e"
server_key_iv: "028ce24fef3893fd6523cda2dcb9675a"
server_key_version: "encryptionkey@0"
I20260810 04:45:46.847774 15317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "c61835810a8543f280a65e37be1bdd46"
format_stamp: "Formatted at 2026-08-10 04:45:46 on dist-test-slave-8hhm"
server_key: "5059fa3dd6f4cb311aefcc523b357f3e"
server_key_iv: "028ce24fef3893fd6523cda2dcb9675a"
server_key_version: "encryptionkey@0"
I20260810 04:45:46.849120 15317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:45:46.849838 15341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:46.850000 15317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:45:46.850139 15317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "c61835810a8543f280a65e37be1bdd46"
format_stamp: "Formatted at 2026-08-10 04:45:46 on dist-test-slave-8hhm"
server_key: "5059fa3dd6f4cb311aefcc523b357f3e"
server_key_iv: "028ce24fef3893fd6523cda2dcb9675a"
server_key_version: "encryptionkey@0"
I20260810 04:45:46.850225 15317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:46.874789 15317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:46.877316 15317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:46.877475 15317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:46.881842 15317 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:36223
I20260810 04:45:46.881947 15430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:36223 every 8 connection(s)
I20260810 04:45:46.882340 15317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:45:46.883605 15431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:46.883760 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 15317
I20260810 04:45:46.883888 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:45:46.884757 15284 external_mini_cluster.cc:1428] Setting key 7a73d017fcdee11b30c5e678111f5514
I20260810 04:45:46.887111 15431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46: Bootstrap starting.
I20260810 04:45:46.887820 15431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:46.888152 15431 log.cc:824] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:46.889334 15431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46: No bootstrap required, opened a new log
I20260810 04:45:46.892060 15431 raft_consensus.cc:348] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61835810a8543f280a65e37be1bdd46" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36223 } }
I20260810 04:45:46.892196 15431 raft_consensus.cc:374] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:46.892220 15431 raft_consensus.cc:729] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61835810a8543f280a65e37be1bdd46, State: Initialized, Role: FOLLOWER
I20260810 04:45:46.892354 15431 consensus_queue.cc:260] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61835810a8543f280a65e37be1bdd46" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36223 } }
I20260810 04:45:46.892436 15431 raft_consensus.cc:388] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:45:46.892478 15431 raft_consensus.cc:482] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:45:46.892529 15431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:46.893151 15431 raft_consensus.cc:504] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61835810a8543f280a65e37be1bdd46" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36223 } }
I20260810 04:45:46.893280 15431 leader_election.cc:302] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c61835810a8543f280a65e37be1bdd46; no voters:
I20260810 04:45:46.893432 15431 leader_election.cc:288] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:45:46.893478 15438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:46.893637 15438 raft_consensus.cc:686] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [term 1 LEADER]: Becoming Leader. State: Replica: c61835810a8543f280a65e37be1bdd46, State: Running, Role: LEADER
I20260810 04:45:46.893707 15431 sys_catalog.cc:564] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:45:46.893778 15438 consensus_queue.cc:237] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61835810a8543f280a65e37be1bdd46" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36223 } }
I20260810 04:45:46.894200 15441 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c61835810a8543f280a65e37be1bdd46. Latest consensus state: current_term: 1 leader_uuid: "c61835810a8543f280a65e37be1bdd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61835810a8543f280a65e37be1bdd46" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36223 } } }
I20260810 04:45:46.894204 15440 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c61835810a8543f280a65e37be1bdd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c61835810a8543f280a65e37be1bdd46" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36223 } } }
I20260810 04:45:46.894343 15441 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [sys.catalog]: This master's current role is: LEADER
I20260810 04:45:46.894348 15440 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46 [sys.catalog]: This master's current role is: LEADER
I20260810 04:45:46.895188 15451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:45:46.895782 15451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:45:46.897295 15451 catalog_manager.cc:1347] Generated new cluster ID: a2652d36ebd0436bade63db8f9ecc1d7
I20260810 04:45:46.897349 15451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:45:46 dist-test-slave-8hhm krb5kdc[15299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:46.902438 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:46.885859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.14.237.62:36223)
0810 04:45:46.886034 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:46.886043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:46.886324 (+ 281us) server_negotiation.cc:407] Connection header received
0810 04:45:46.889224 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:46.889233 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:46.889315 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:46.889456 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:46.891701 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:46.892815 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:46.894451 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:46.895061 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:46.895149 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:46.898919 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:46.898923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:46.898923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:46.898937 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:46.900570 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:46.901017 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:46.901020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:46.901020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:46.901107 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:46.901435 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:46.901436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:46.901436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:46.901654 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:46.901839 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:46.902116 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 04:45:46.902259 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260810 04:45:46.905392 15451 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:45:46.906039 15451 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:45:46.911451 15451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c61835810a8543f280a65e37be1bdd46: Generated new TSK 0
I20260810 04:45:46.911607 15451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:45:46.942211 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:36223
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:47.022758 15468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:47.023005 15468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:47.023046 15468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:47.024246 15468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:47.024315 15468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:47.024361 15468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:45:47.025863 15468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:36223
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:47.026147 15468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:47.026367 15468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:45:47.031605 15468 server_base.cc:1034] running on GCE node
W20260810 04:45:47.031551 15476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:47.031576 15473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:47.031584 15483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:47.031952 15468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:47.032238 15468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:47.033401 15468 hybrid_clock.cc:648] HybridClock initialized: now 1786337147033396 us; error 36 us; skew 500 ppm
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:47.035909 15468 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:45:47.036321 15468 webserver.cc:460] Webserver started at http://127.14.237.1:39617/ using document root <none> and password file <none>
I20260810 04:45:47.036504 15468 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:47.036542 15468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:47.036628 15468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:47.037686 15468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "c1828322367b41dc9b00dd15d5c743a4"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "8695fe984a6c8e016f11575f5d0c9dd9"
server_key_iv: "cae6a664b9e32a952e602b8b220f10ed"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.037984 15468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "c1828322367b41dc9b00dd15d5c743a4"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "8695fe984a6c8e016f11575f5d0c9dd9"
server_key_iv: "cae6a664b9e32a952e602b8b220f10ed"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.039204 15468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:45:47.039878 15494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.040086 15468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:47.040172 15468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "c1828322367b41dc9b00dd15d5c743a4"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "8695fe984a6c8e016f11575f5d0c9dd9"
server_key_iv: "cae6a664b9e32a952e602b8b220f10ed"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.040230 15468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:47.050086 15468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:47.052345 15468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:47.052448 15468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:47.052752 15468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:45:47.053092 15468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:47.053128 15468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.053148 15468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:47.053160 15468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.059036 15468 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:45779
I20260810 04:45:47.059108 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:45779 every 8 connection(s)
I20260810 04:45:47.059443 15468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:47.067430 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 15468
I20260810 04:45:47.067523 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:45:47.067691 15284 external_mini_cluster.cc:1428] Setting key acbfd4b26046a42b453b7d757726b7f3
I20260810 04:45:47.068785 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:47.060188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:45923 (local address 127.14.237.62:36223)
0810 04:45:47.060268 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:47.060270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:47.060476 (+ 206us) server_negotiation.cc:407] Connection header received
0810 04:45:47.061004 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:47.061005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:47.061017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:47.061077 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:47.062195 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.062690 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.063629 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.063826 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.063882 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:47.065985 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:47.065986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:47.065986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:47.065993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:47.067199 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.067693 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.067694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.067694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.067781 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.068103 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.068105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.068105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.068228 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:47.068308 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:47.068587 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 04:45:47.068683 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:45:47.068924 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:47.060165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:36223 (local address 127.14.237.1:45923)
0810 04:45:47.060354 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:47.060389 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:47.060878 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:47.061130 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:47.061133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:47.061905 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:47.062091 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:47.062096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.062778 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.062778 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:47.063534 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:47.063536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.063904 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.063905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:47.064007 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:47.064451 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:47.064459 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:47.065900 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:47.067282 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:47.067289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:47.067291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:47.067616 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:47.067868 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:47.067869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:47.067869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:47.068025 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:47.068311 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:47.068319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:47.068453 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 04:45:47.068553 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 04:45:47.068704 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":111,"threads_started":1}
I20260810 04:45:47.069428 15667 heartbeater.cc:347] Connected to a master server at 127.14.237.62:36223
I20260810 04:45:47.069531 15667 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:47.069715 15667 heartbeater.cc:511] Master 127.14.237.62:36223 requested a full tablet report, sending...
I20260810 04:45:47.070218 15366 ts_manager.cc:194] Registered new tserver with Master: c1828322367b41dc9b00dd15d5c743a4 (127.14.237.1:45779)
I20260810 04:45:47.070858 15366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:45923
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:45:47.092183 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:36223
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:47.173827 15678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:47.174052 15678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:47.174088 15678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:47.175170 15678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:47.175239 15678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:47.175284 15678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:45:47.176730 15678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:36223
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:47.176988 15678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:47.177259 15678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:47.182428 15689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:47.182447 15692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:47.182483 15687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:47.182587 15678 server_base.cc:1034] running on GCE node
I20260810 04:45:47.182862 15678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:47.183118 15678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:47.184284 15678 hybrid_clock.cc:648] HybridClock initialized: now 1786337147184276 us; error 30 us; skew 500 ppm
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:47.186748 15678 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:45:47.187146 15678 webserver.cc:460] Webserver started at http://127.14.237.2:46493/ using document root <none> and password file <none>
I20260810 04:45:47.187351 15678 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:47.187415 15678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:47.187570 15678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:47.188778 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "60546f650b7c46b993b7c211fe4017b9"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "976b8e0a39dff7cbb5cf60689d0fad59"
server_key_iv: "38c060a7b2d4bcd1b403c3e90121b826"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.189157 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "60546f650b7c46b993b7c211fe4017b9"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "976b8e0a39dff7cbb5cf60689d0fad59"
server_key_iv: "38c060a7b2d4bcd1b403c3e90121b826"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.190490 15678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:47.191247 15699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.191430 15678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:47.191531 15678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "60546f650b7c46b993b7c211fe4017b9"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "976b8e0a39dff7cbb5cf60689d0fad59"
server_key_iv: "38c060a7b2d4bcd1b403c3e90121b826"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.191619 15678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:47.216970 15678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:47.219187 15678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:47.219360 15678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:47.219614 15678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:45:47.219945 15678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:47.220000 15678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.220036 15678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:47.220062 15678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.226052 15678 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:45211
I20260810 04:45:47.226161 15887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:45211 every 8 connection(s)
I20260810 04:45:47.226464 15678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:45:47.227164 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 15678
I20260810 04:45:47.227270 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:45:47.227454 15284 external_mini_cluster.cc:1428] Setting key bd41a42013f5dde19fe54a42b7258773
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:47.237424 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:47.227276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:48413 (local address 127.14.237.62:36223)
0810 04:45:47.227327 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:47.227329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:47.227620 (+ 291us) server_negotiation.cc:407] Connection header received
0810 04:45:47.228290 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:47.228292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:47.228305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:47.228378 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:47.229317 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.230083 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.231576 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.231762 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.231830 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:47.234295 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:47.234297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:47.234297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:47.234306 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:47.235924 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.236329 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.236330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.236331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.236396 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.236697 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.236699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.236699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.236866 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:47.236948 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:47.237244 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 04:45:47.237332 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
I20260810 04:45:47.237540 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:47.227272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:36223 (local address 127.14.237.2:48413)
0810 04:45:47.227447 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:47.227494 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:47.228184 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:47.228443 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:47.228446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:47.228994 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:47.229175 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:47.229180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.230173 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.230174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:47.231479 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:47.231483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.231852 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.231853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:47.232006 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:47.232507 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:47.232517 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:47.234201 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:47.236014 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:47.236020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:47.236021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:47.236246 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:47.236466 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:47.236467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:47.236467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:47.236610 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:47.236939 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:47.236943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:47.237052 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 04:45:47.237144 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 04:45:47.237350 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260810 04:45:47.238118 15889 heartbeater.cc:347] Connected to a master server at 127.14.237.62:36223
I20260810 04:45:47.238200 15889 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:47.238336 15889 heartbeater.cc:511] Master 127.14.237.62:36223 requested a full tablet report, sending...
I20260810 04:45:47.238713 15366 ts_manager.cc:194] Registered new tserver with Master: 60546f650b7c46b993b7c211fe4017b9 (127.14.237.2:45211)
I20260810 04:45:47.239147 15366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:48413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:45:47.256103 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:36223
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:47.337472 15901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:47.337658 15901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:47.337678 15901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:47.338752 15901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:47.338794 15901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:47.338845 15901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:45:47.340250 15901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:36223
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:47.340484 15901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:47.340734 15901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:47.345988 15911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:47.346006 15915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:47.346094 15901 server_base.cc:1034] running on GCE node
W20260810 04:45:47.345985 15910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:47.346517 15901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:47.346812 15901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:47.348002 15901 hybrid_clock.cc:648] HybridClock initialized: now 1786337147347983 us; error 28 us; skew 500 ppm
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:47.350728 15901 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:45:47.351245 15901 webserver.cc:460] Webserver started at http://127.14.237.3:33575/ using document root <none> and password file <none>
I20260810 04:45:47.351492 15901 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:47.351576 15901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:47.351742 15901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:47.352906 15901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "a2d61ee7b4b84ff5b54a5820a62f0082"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "df6af9ee303d7e53183916b9d793bfee"
server_key_iv: "92323eed67e0329256e3988a058bd42e"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.353365 15901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "a2d61ee7b4b84ff5b54a5820a62f0082"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "df6af9ee303d7e53183916b9d793bfee"
server_key_iv: "92323eed67e0329256e3988a058bd42e"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.354815 15901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:47.355558 15929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.355741 15901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:47.355855 15901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "a2d61ee7b4b84ff5b54a5820a62f0082"
format_stamp: "Formatted at 2026-08-10 04:45:47 on dist-test-slave-8hhm"
server_key: "df6af9ee303d7e53183916b9d793bfee"
server_key_iv: "92323eed67e0329256e3988a058bd42e"
server_key_version: "encryptionkey@0"
I20260810 04:45:47.355934 15901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:47.365387 15901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:47.367209 15901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:47.367381 15901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:47.367619 15901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:45:47.367956 15901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:47.368026 15901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.368065 15901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:47.368098 15901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:47.374552 15901 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:32813
I20260810 04:45:47.374580 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:32813 every 8 connection(s)
I20260810 04:45:47.374908 15901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:47.381543 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 15901
I20260810 04:45:47.381667 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:45:47.381858 15284 external_mini_cluster.cc:1428] Setting key f540d3c41a17547932133c93fdb995c4
I20260810 04:45:47.384382 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:47.375779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:54253 (local address 127.14.237.62:36223)
0810 04:45:47.375845 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:47.375851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:47.376046 (+ 195us) server_negotiation.cc:407] Connection header received
0810 04:45:47.376514 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:47.376515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:47.376542 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:47.376598 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:47.377628 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.378080 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.379019 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.379210 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.379265 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:47.381403 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:47.381405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:47.381405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:47.381412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:47.383153 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.383549 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.383549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.383550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.383602 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.383821 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.383821 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.383822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.383926 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:47.383991 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:47.384254 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 04:45:47.384320 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:45:47.384591 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:47.375743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:36223 (local address 127.14.237.3:54253)
0810 04:45:47.375912 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:47.375952 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:47.376436 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:47.376652 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:47.376655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:47.377354 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:47.377533 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:47.377539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.378188 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.378189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:47.378916 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:47.378918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.379286 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.379287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:47.379381 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:47.379794 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:47.379802 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:47.381174 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:47.383238 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:47.383244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:47.383246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:47.383481 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:47.383663 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:47.383664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:47.383664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:47.383764 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:47.384009 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:47.384014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:47.384138 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 04:45:47.384229 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:45:47.384346 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260810 04:45:47.385070 16113 heartbeater.cc:347] Connected to a master server at 127.14.237.62:36223
I20260810 04:45:47.385164 16113 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:47.385358 16113 heartbeater.cc:511] Master 127.14.237.62:36223 requested a full tablet report, sending...
I20260810 04:45:47.385687 15366 ts_manager.cc:194] Registered new tserver with Master: a2d61ee7b4b84ff5b54a5820a62f0082 (127.14.237.3:32813)
I20260810 04:45:47.385998 15366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:54253
I20260810 04:45:47.393891 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:45:47.418429 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:47.410492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.14.237.1:45779)
0810 04:45:47.410651 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:47.410656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:47.410667 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:45:47.410717 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:47.410719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:47.410733 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:47.410806 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:47.411395 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.412265 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.412962 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.413322 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.413378 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:47.415639 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:47.415648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:47.415648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:47.415654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:47.417339 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.417673 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.417676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.417676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.417733 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.417973 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.417973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.417973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.418101 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:47.418213 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:47.418293 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 04:45:47.418354 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260810 04:45:47.418834 15537 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35086
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[15299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:45:47.441905 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:47.434900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.14.237.1:45779)
0810 04:45:47.434980 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:47.434981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:47.435022 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:45:47.435097 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:47.435098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:47.435129 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:47.435186 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:47.435751 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.436266 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.436926 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:47.437140 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:47.437191 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:47.439362 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:47.439363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:47.439363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:47.439370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:47.440746 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.441078 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.441079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.441079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.441140 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:47.441432 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:47.441432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:47.441433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:47.441531 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:47.441638 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:47.441710 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 04:45:47.441777 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 04:45:47.442481 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 15468
I20260810 04:45:47.456496 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 15678
I20260810 04:45:47.466742 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 15901
I20260810 04:45:47.475631 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 15317
2026-08-10T04:45:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2122 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[16137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[16137](info): set up 2 sockets
Aug 10 04:45:47 dist-test-slave-8hhm krb5kdc[16137](info): commencing operation
krb5kdc: starting...
W20260810 04:45:48.716369 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:45:48 dist-test-slave-8hhm krb5kdc[16137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:45:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:45:48.837862 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:45553
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:45553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:48.922488 16154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:48.922713 16154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:48.922757 16154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:48.923800 16154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:45:48.923863 16154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:48.923885 16154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:48.923904 16154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:45:48.923933 16154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:45:48.925333 16154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:45553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:48.925595 16154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:48.925812 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:48.930540 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:48.930579 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:48.930586 16168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:48.930913 16154 server_base.cc:1034] running on GCE node
I20260810 04:45:48.931166 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:48.931457 16154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:48.932619 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786337148932613 us; error 36 us; skew 500 ppm
Aug 10 04:45:48 dist-test-slave-8hhm krb5kdc[16137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337148, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:48.934996 16154 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:45:48.935439 16154 webserver.cc:460] Webserver started at http://127.14.237.62:46327/ using document root <none> and password file <none>
I20260810 04:45:48.935703 16154 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:48.935773 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:48.935925 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:48.936928 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "7d29319a76374b1f852c36dbb9cfa153"
format_stamp: "Formatted at 2026-08-10 04:45:48 on dist-test-slave-8hhm"
server_key: "853156356c912443197ef725a3bc689d"
server_key_iv: "2ea15d0f69966c93e5dd7931d6c37f36"
server_key_version: "encryptionkey@0"
I20260810 04:45:48.937353 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "7d29319a76374b1f852c36dbb9cfa153"
format_stamp: "Formatted at 2026-08-10 04:45:48 on dist-test-slave-8hhm"
server_key: "853156356c912443197ef725a3bc689d"
server_key_iv: "2ea15d0f69966c93e5dd7931d6c37f36"
server_key_version: "encryptionkey@0"
I20260810 04:45:48.938705 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:48.939426 16183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:48.939595 16154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:48.939690 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "7d29319a76374b1f852c36dbb9cfa153"
format_stamp: "Formatted at 2026-08-10 04:45:48 on dist-test-slave-8hhm"
server_key: "853156356c912443197ef725a3bc689d"
server_key_iv: "2ea15d0f69966c93e5dd7931d6c37f36"
server_key_version: "encryptionkey@0"
I20260810 04:45:48.939774 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:48.979839 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:48.981920 16154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:48.982112 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:48.986061 16154 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:45553
I20260810 04:45:48.986100 16254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:45553 every 8 connection(s)
I20260810 04:45:48.986506 16154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:45:48.987377 16255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:48.989622 16255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153: Bootstrap starting.
I20260810 04:45:48.990165 16255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:48.990399 16255 log.cc:824] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:48.991047 16255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153: No bootstrap required, opened a new log
I20260810 04:45:48.992664 16255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29319a76374b1f852c36dbb9cfa153" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45553 } }
I20260810 04:45:48.992803 16255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:48.992770 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16154
I20260810 04:45:48.992826 16255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d29319a76374b1f852c36dbb9cfa153, State: Initialized, Role: FOLLOWER
I20260810 04:45:48.992867 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:45:48.992982 16255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29319a76374b1f852c36dbb9cfa153" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45553 } }
I20260810 04:45:48.993057 15284 external_mini_cluster.cc:1428] Setting key af1b7c1f46bb0e693354dd0f899642b7
I20260810 04:45:48.993057 16255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:45:48.993103 16255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:45:48.993151 16255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:48.994222 16255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29319a76374b1f852c36dbb9cfa153" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45553 } }
I20260810 04:45:48.994344 16255 leader_election.cc:302] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d29319a76374b1f852c36dbb9cfa153; no voters:
I20260810 04:45:48.994513 16255 leader_election.cc:288] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:45:48.994565 16265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:48.994707 16265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [term 1 LEADER]: Becoming Leader. State: Replica: 7d29319a76374b1f852c36dbb9cfa153, State: Running, Role: LEADER
I20260810 04:45:48.994781 16255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:45:48.994803 16265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29319a76374b1f852c36dbb9cfa153" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45553 } }
I20260810 04:45:48.995107 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d29319a76374b1f852c36dbb9cfa153. Latest consensus state: current_term: 1 leader_uuid: "7d29319a76374b1f852c36dbb9cfa153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29319a76374b1f852c36dbb9cfa153" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45553 } } }
I20260810 04:45:48.995206 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [sys.catalog]: This master's current role is: LEADER
I20260810 04:45:48.995117 16266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d29319a76374b1f852c36dbb9cfa153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d29319a76374b1f852c36dbb9cfa153" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45553 } } }
I20260810 04:45:48.995368 16266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153 [sys.catalog]: This master's current role is: LEADER
I20260810 04:45:48.995622 16274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:45:48.996140 16274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:45:48.997787 16274 catalog_manager.cc:1347] Generated new cluster ID: ba5dfe92248648599d36df19abf6d7a6
I20260810 04:45:48.997836 16274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:45:48 dist-test-slave-8hhm krb5kdc[16137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:49.004490 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:48.993556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.14.237.62:45553)
0810 04:45:48.993796 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:48.993803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:48.993830 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:45:48.993920 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:48.993926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:48.993964 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:48.994067 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:48.994911 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:48.996263 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:48.997421 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:48.997929 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:48.998035 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.000646 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:49.000650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:49.000651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:49.000662 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:49.002677 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.003095 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.003097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.003098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.003219 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.003442 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.003442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.003443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.003713 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:49.003873 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.004125 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.004268 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":160,"threads_started":1}
I20260810 04:45:49.006681 16274 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:45:49.007121 16274 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:45:49.016831 16274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d29319a76374b1f852c36dbb9cfa153: Generated new TSK 0
I20260810 04:45:49.017004 16274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:45:49.048454 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45553
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:49.130832 16301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:49.131042 16301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:49.131065 16301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:49.132156 16301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:49.132193 16301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:49.132220 16301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:45:49.133831 16301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:49.134011 16301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:49.134181 16301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:49.139037 16309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:49.139084 16314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:49.139122 16308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:49.139442 16301 server_base.cc:1034] running on GCE node
I20260810 04:45:49.139669 16301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:49.139900 16301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:49.141064 16301 hybrid_clock.cc:648] HybridClock initialized: now 1786337149141062 us; error 36 us; skew 500 ppm
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:49.143888 16301 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:45:49.144323 16301 webserver.cc:460] Webserver started at http://127.14.237.1:42817/ using document root <none> and password file <none>
I20260810 04:45:49.144567 16301 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:49.144613 16301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:49.144776 16301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:49.145897 16301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "cef743a355e04298af74ac72020e58d7"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "7b8789c53347b01ba55eacbc4b02e9e8"
server_key_iv: "c6566e033b5f93b11ab014dd576bc066"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.146279 16301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "cef743a355e04298af74ac72020e58d7"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "7b8789c53347b01ba55eacbc4b02e9e8"
server_key_iv: "c6566e033b5f93b11ab014dd576bc066"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.147650 16301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:45:49.148402 16324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.148640 16301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:49.148718 16301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "cef743a355e04298af74ac72020e58d7"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "7b8789c53347b01ba55eacbc4b02e9e8"
server_key_iv: "c6566e033b5f93b11ab014dd576bc066"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.148770 16301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:49.189428 16301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:49.191377 16301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:49.191519 16301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:49.191788 16301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:45:49.192126 16301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:49.192160 16301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.192183 16301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:49.192231 16301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.198343 16301 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:46573
I20260810 04:45:49.198393 16495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:46573 every 8 connection(s)
I20260810 04:45:49.198702 16301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:45:49.204047 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16301
I20260810 04:45:49.204183 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:45:49.204350 15284 external_mini_cluster.cc:1428] Setting key 51ada3ef196d9a318f7486966128c3c2
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:49.208593 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.199400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:51429 (local address 127.14.237.62:45553)
0810 04:45:49.199449 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.199451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.199667 (+ 216us) server_negotiation.cc:407] Connection header received
0810 04:45:49.200672 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.200673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.200684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.200751 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:49.201479 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.201985 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.203011 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.203256 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.203317 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.205713 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:49.205714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:49.205714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:49.205721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:49.207005 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.207490 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.207491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.207492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.207543 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.207808 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.207809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.207809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.207904 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:49.207952 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.208355 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.208447 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:45:49.208608 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.199350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:45553 (local address 127.14.237.1:51429)
0810 04:45:49.199522 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:49.199565 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:49.200566 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:49.200838 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:49.200842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:49.201109 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:49.201383 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.201389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.202069 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.202069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.202922 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.202924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.203338 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.203338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.203438 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.203989 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:49.203998 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:49.205626 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:49.207084 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:49.207092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:49.207094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:49.207412 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:49.207607 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:49.207608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:49.207608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:49.207745 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:49.207985 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:49.207991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:49.208157 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 04:45:49.208327 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 04:45:49.208461 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260810 04:45:49.209115 16496 heartbeater.cc:347] Connected to a master server at 127.14.237.62:45553
I20260810 04:45:49.209232 16496 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:49.209415 16496 heartbeater.cc:511] Master 127.14.237.62:45553 requested a full tablet report, sending...
I20260810 04:45:49.209944 16209 ts_manager.cc:194] Registered new tserver with Master: cef743a355e04298af74ac72020e58d7 (127.14.237.1:46573)
I20260810 04:45:49.210542 16209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:51429
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:45:49.230280 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45553
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:49.311026 16505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:49.311269 16505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:49.311308 16505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:49.312463 16505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:49.312532 16505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:49.312577 16505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:45:49.314101 16505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:49.314357 16505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:49.314580 16505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:49.319645 16519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:49.319692 16516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:49.319693 16515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:49.319736 16505 server_base.cc:1034] running on GCE node
I20260810 04:45:49.320029 16505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:49.320302 16505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:49.321465 16505 hybrid_clock.cc:648] HybridClock initialized: now 1786337149321450 us; error 24 us; skew 500 ppm
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:49.324149 16505 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:45:49.324625 16505 webserver.cc:460] Webserver started at http://127.14.237.2:40955/ using document root <none> and password file <none>
I20260810 04:45:49.324841 16505 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:49.324887 16505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:49.325042 16505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:49.326120 16505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "c00e800155f14a308990508ad4e7d6ee"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "f8446aef69599e25d37e5f14e09f9aa7"
server_key_iv: "60aa20a382355d7d1dc3a4d5aec014aa"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.326478 16505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "c00e800155f14a308990508ad4e7d6ee"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "f8446aef69599e25d37e5f14e09f9aa7"
server_key_iv: "60aa20a382355d7d1dc3a4d5aec014aa"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.327800 16505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:45:49.328593 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.328779 16505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:49.328892 16505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "c00e800155f14a308990508ad4e7d6ee"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "f8446aef69599e25d37e5f14e09f9aa7"
server_key_iv: "60aa20a382355d7d1dc3a4d5aec014aa"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.328967 16505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:49.351869 16505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:49.354276 16505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:49.354398 16505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:49.354632 16505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:45:49.355013 16505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:49.355047 16505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.355094 16505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:49.355125 16505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.360960 16505 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:37693
I20260810 04:45:49.361004 16711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:37693 every 8 connection(s)
I20260810 04:45:49.361404 16505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:45:49.365417 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16505
I20260810 04:45:49.365527 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:45:49.365702 15284 external_mini_cluster.cc:1428] Setting key d26e40c54373b40ff954753ecab5b08d
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:49.371178 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.362152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:49033 (local address 127.14.237.62:45553)
0810 04:45:49.362204 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.362207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.362397 (+ 190us) server_negotiation.cc:407] Connection header received
0810 04:45:49.363299 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.363300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.363309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.363372 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:49.363960 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.364446 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.365495 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.365792 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.365871 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.368262 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:49.368263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:49.368264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:49.368271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:49.369742 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.370176 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.370190 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.370191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.370254 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.370512 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.370512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.370513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.370644 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:49.370717 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.370996 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.371090 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:45:49.371371 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.362097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:45553 (local address 127.14.237.2:49033)
0810 04:45:49.362258 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:49.362301 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:49.363209 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:49.363429 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:49.363432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:49.363692 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:49.363883 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.363887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.364531 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.364532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.365398 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.365401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.365894 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.365896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.366067 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.366680 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:49.366688 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:49.368151 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:49.369837 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:49.369843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:49.369845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:49.370063 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:49.370322 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:49.370323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:49.370323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:49.370436 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:49.370719 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:49.370726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:49.370866 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 04:45:49.370975 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 04:45:49.371131 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260810 04:45:49.371850 16713 heartbeater.cc:347] Connected to a master server at 127.14.237.62:45553
I20260810 04:45:49.371941 16713 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:49.372150 16713 heartbeater.cc:511] Master 127.14.237.62:45553 requested a full tablet report, sending...
I20260810 04:45:49.372555 16209 ts_manager.cc:194] Registered new tserver with Master: c00e800155f14a308990508ad4e7d6ee (127.14.237.2:37693)
I20260810 04:45:49.373059 16209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:49033
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:45:49.392061 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45553
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:49.477450 16725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:49.477632 16725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:49.477654 16725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:49.478679 16725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:49.478711 16725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:49.478736 16725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:45:49.480026 16725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:49.480193 16725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:49.480356 16725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:49.485342 16743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:49.485409 16736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:49.485461 16725 server_base.cc:1034] running on GCE node
W20260810 04:45:49.485432 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:49.485711 16725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:49.485988 16725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:49.487133 16725 hybrid_clock.cc:648] HybridClock initialized: now 1786337149487113 us; error 37 us; skew 500 ppm
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:49.489769 16725 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:45:49.490234 16725 webserver.cc:460] Webserver started at http://127.14.237.3:35583/ using document root <none> and password file <none>
I20260810 04:45:49.490483 16725 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:49.490553 16725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:49.490705 16725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:49.491717 16725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "724f3a19d92449efaf32c439e770886d"
server_key_iv: "4bc4ea46ecdab55a0d1435e3db300de1"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.492098 16725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "724f3a19d92449efaf32c439e770886d"
server_key_iv: "4bc4ea46ecdab55a0d1435e3db300de1"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.493499 16725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:45:49.494240 16756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.494443 16725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:49.494544 16725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3"
format_stamp: "Formatted at 2026-08-10 04:45:49 on dist-test-slave-8hhm"
server_key: "724f3a19d92449efaf32c439e770886d"
server_key_iv: "4bc4ea46ecdab55a0d1435e3db300de1"
server_key_version: "encryptionkey@0"
I20260810 04:45:49.494638 16725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:49.514446 16725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:49.516847 16725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:49.517023 16725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:49.517341 16725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:45:49.517671 16725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:49.517737 16725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.517779 16725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:49.517807 16725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:49.523505 16725 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:32969
I20260810 04:45:49.523550 16958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:32969 every 8 connection(s)
I20260810 04:45:49.523862 16725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:45:49.527132 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16725
I20260810 04:45:49.527236 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:45:49.527380 15284 external_mini_cluster.cc:1428] Setting key 58651033f30e63c58518ee13cd5aa247
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:49.533881 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.524558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:45843 (local address 127.14.237.62:45553)
0810 04:45:49.524626 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.524628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.524814 (+ 186us) server_negotiation.cc:407] Connection header received
0810 04:45:49.525844 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.525845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.525859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.525908 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:49.526543 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.527020 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.528336 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.528552 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.528610 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.530783 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:49.530784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:49.530785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:49.530791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:49.532381 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.532854 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.532855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.532855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.532922 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.533203 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.533204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.533205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.533352 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:49.533425 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.533711 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.533794 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:45:49.534055 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.524541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:45553 (local address 127.14.237.3:45843)
0810 04:45:49.524689 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:49.524730 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:49.525722 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:49.525983 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:49.525986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:49.526247 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:49.526430 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.526435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.527125 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.527126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.528253 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.528256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.528657 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.528658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.528764 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.529209 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:49.529266 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:49.530703 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:49.532467 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:49.532473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:49.532474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:49.532752 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:49.533010 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:49.533011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:49.533011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:49.533124 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:49.533428 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:49.533435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:49.533559 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 04:45:49.533674 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 04:45:49.533826 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260810 04:45:49.534551 16961 heartbeater.cc:347] Connected to a master server at 127.14.237.62:45553
I20260810 04:45:49.534664 16961 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:49.534868 16961 heartbeater.cc:511] Master 127.14.237.62:45553 requested a full tablet report, sending...
I20260810 04:45:49.535265 16209 ts_manager.cc:194] Registered new tserver with Master: 8d07e0e6dedb4fd3a5e062c3c54503f3 (127.14.237.3:32969)
I20260810 04:45:49.535633 16209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:45843
I20260810 04:45:49.539126 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:45:49.546617 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.540718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.14.237.62:45553)
0810 04:45:49.540775 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.540779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.540897 (+ 118us) server_negotiation.cc:407] Connection header received
0810 04:45:49.540935 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.540935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.540944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.540998 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:49.541574 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.542103 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.542745 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.542942 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.543032 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.544084 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:49.544086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:49.544086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:49.544110 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:49.545569 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.545875 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.545875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.545876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.545927 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.546137 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.546139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.546139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.546300 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:49.546409 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.546495 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.546555 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:45:49.548738 16209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:45:49.550398 16209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:49.556957 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.553045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:37693 (local address 127.0.0.1:56036)
0810 04:45:49.553524 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:49.553535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:49.553594 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:49.553861 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:49.553862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:49.553864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:49.553906 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.553908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.555106 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.555111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.555962 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.555964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.556642 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.556642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.556720 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.556733 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:45:49.556833 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 04:45:49.556885 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":107,"threads_started":1}
I20260810 04:45:49.556972 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.553336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.14.237.2:37693)
0810 04:45:49.553533 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.553537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.553583 (+ 46us) server_negotiation.cc:407] Connection header received
0810 04:45:49.553665 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.553667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.553684 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.553760 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:49.553980 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.554995 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.556055 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.556532 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.556637 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.556701 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.556854 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.556897 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260810 04:45:49.557305 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.552916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:46573 (local address 127.0.0.1:41438)
0810 04:45:49.553112 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:49.553155 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:49.553252 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:49.553619 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:49.553622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:49.553626 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:49.553692 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.553694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.555052 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.555053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.556078 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.556081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.556919 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.556920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.556991 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.556997 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:45:49.557162 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 04:45:49.557202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":57,"threads_started":1}
I20260810 04:45:49.557495 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.553157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.14.237.3:32969)
0810 04:45:49.553561 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.553566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.553576 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:45:49.553616 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.553618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.553703 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.553806 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:49.554010 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.554663 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.556096 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.556798 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.556872 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.556931 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.557352 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.557401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":115,"threads_started":1}
I20260810 04:45:49.557631 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.552916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:32969 (local address 127.0.0.1:46510)
0810 04:45:49.553367 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:49.553378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:49.553437 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:49.553899 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:49.553900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:49.553902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:49.553950 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.553951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.554774 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.554775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.555979 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.555982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.556958 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.556959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.557009 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.557013 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:45:49.557256 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 04:45:49.557290 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260810 04:45:49.557711 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.553167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.14.237.1:46573)
0810 04:45:49.553401 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.553405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.553413 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:45:49.553472 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.553474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.553491 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.553775 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:49.553871 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.554946 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.556161 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.556826 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.557485 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.557556 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.557584 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.557625 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":114,"threads_started":1}
I20260810 04:45:49.557765 16613 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e9e8d52a54a87b0e86623ef259791 (DEFAULT_TABLE table=test-table [id=7df3a1999f124ba9a73eae6453adefff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:49.558166 16613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e9e8d52a54a87b0e86623ef259791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:49.558334 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e9e8d52a54a87b0e86623ef259791 (DEFAULT_TABLE table=test-table [id=7df3a1999f124ba9a73eae6453adefff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:49.558662 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e9e8d52a54a87b0e86623ef259791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:49.560702 16990 tablet_bootstrap.cc:492] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee: Bootstrap starting.
I20260810 04:45:49.561133 16992 tablet_bootstrap.cc:492] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7: Bootstrap starting.
I20260810 04:45:49.561445 16990 tablet_bootstrap.cc:654] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:49.561599 16842 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e9e8d52a54a87b0e86623ef259791 (DEFAULT_TABLE table=test-table [id=7df3a1999f124ba9a73eae6453adefff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:49.561853 16992 tablet_bootstrap.cc:654] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:49.561968 16842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e9e8d52a54a87b0e86623ef259791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:49.562079 16990 log.cc:824] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:49.562465 16992 log.cc:824] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:49.562912 16990 tablet_bootstrap.cc:492] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee: No bootstrap required, opened a new log
I20260810 04:45:49.562985 16990 ts_tablet_manager.cc:1397] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:45:49.563184 16992 tablet_bootstrap.cc:492] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7: No bootstrap required, opened a new log
I20260810 04:45:49.563266 16992 ts_tablet_manager.cc:1397] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:45:49.564754 16992 raft_consensus.cc:348] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.564913 16992 raft_consensus.cc:374] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:49.564956 16992 raft_consensus.cc:729] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef743a355e04298af74ac72020e58d7, State: Initialized, Role: FOLLOWER
I20260810 04:45:49.564996 16990 raft_consensus.cc:348] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.565084 16992 consensus_queue.cc:260] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.565163 16990 raft_consensus.cc:374] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:49.565228 16990 raft_consensus.cc:729] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00e800155f14a308990508ad4e7d6ee, State: Initialized, Role: FOLLOWER
I20260810 04:45:49.565366 16997 tablet_bootstrap.cc:492] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3: Bootstrap starting.
I20260810 04:45:49.565353 16990 consensus_queue.cc:260] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.565402 16992 ts_tablet_manager.cc:1428] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:45:49.565528 16496 heartbeater.cc:503] Master 127.14.237.62:45553 was elected leader, sending a full tablet report...
I20260810 04:45:49.565660 16990 ts_tablet_manager.cc:1428] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:45:49.565681 16713 heartbeater.cc:503] Master 127.14.237.62:45553 was elected leader, sending a full tablet report...
I20260810 04:45:49.565987 16997 tablet_bootstrap.cc:654] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:49.566593 16997 log.cc:824] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:49.567387 16997 tablet_bootstrap.cc:492] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3: No bootstrap required, opened a new log
I20260810 04:45:49.567440 16997 ts_tablet_manager.cc:1397] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:45:49.568845 16997 raft_consensus.cc:348] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.568971 16997 raft_consensus.cc:374] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:49.568993 16997 raft_consensus.cc:729] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d07e0e6dedb4fd3a5e062c3c54503f3, State: Initialized, Role: FOLLOWER
I20260810 04:45:49.569100 16997 consensus_queue.cc:260] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.569422 16997 ts_tablet_manager.cc:1428] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:45:49.569458 16961 heartbeater.cc:503] Master 127.14.237.62:45553 was elected leader, sending a full tablet report...
W20260810 04:45:49.611939 16715 tablet.cc:2369] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:45:49.699441 16498 tablet.cc:2369] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:45:49.774439 16965 tablet.cc:2369] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:45:49.788797 16998 raft_consensus.cc:482] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:49.788923 16998 raft_consensus.cc:504] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.789364 16998 leader_election.cc:288] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d07e0e6dedb4fd3a5e062c3c54503f3 (127.14.237.3:32969), c00e800155f14a308990508ad4e7d6ee (127.14.237.2:37693)
I20260810 04:45:49.793651 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.789615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:37693 (local address 127.14.237.1:35749)
0810 04:45:49.789764 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:49.789772 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:49.789823 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:49.789966 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:49.789967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:49.789969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:49.790104 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.790105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.791109 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.791110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.792518 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.792521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.793330 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.793332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.793433 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.793437 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:45:49.793516 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:45:49.793562 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":65,"thread_start_us":85,"threads_started":1}
I20260810 04:45:49.793679 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.789741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:59353 (local address 127.14.237.3:32969)
0810 04:45:49.789784 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.789786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.789798 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:45:49.789854 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.789854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.789867 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.789935 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:49.790264 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.791019 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.792617 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.793237 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.793314 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.793323 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.793549 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.793607 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:45:49.793704 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.789696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:35749 (local address 127.14.237.2:37693)
0810 04:45:49.789747 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.789749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.789837 (+ 88us) server_negotiation.cc:407] Connection header received
0810 04:45:49.789866 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.789867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.789879 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.789935 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:49.790263 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.791017 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.792617 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.793247 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.793315 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.793323 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.793542 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.793581 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:45:49.793941 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.789555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:32969 (local address 127.14.237.1:59353)
0810 04:45:49.789717 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:49.789729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:49.789794 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:49.789967 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:49.789968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:49.789969 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:49.790098 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.790101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.791108 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.791109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.792518 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:49.792521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.793379 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.793380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:49.793458 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.793463 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:45:49.793794 (+ 331us) client_negotiation.cc:240] Negotiation successful
0810 04:45:49.793843 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":127,"thread_start_us":88,"threads_started":1}
I20260810 04:45:49.794150 16642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e9e8d52a54a87b0e86623ef259791" candidate_uuid: "cef743a355e04298af74ac72020e58d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c00e800155f14a308990508ad4e7d6ee" is_pre_election: true
I20260810 04:45:49.794327 16642 raft_consensus.cc:2454] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cef743a355e04298af74ac72020e58d7 in term 0.
I20260810 04:45:49.794566 16326 leader_election.cc:302] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c00e800155f14a308990508ad4e7d6ee, cef743a355e04298af74ac72020e58d7; no voters:
I20260810 04:45:49.794737 16998 raft_consensus.cc:2781] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:49.794795 16998 raft_consensus.cc:482] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:49.794740 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e9e8d52a54a87b0e86623ef259791" candidate_uuid: "cef743a355e04298af74ac72020e58d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" is_pre_election: true
I20260810 04:45:49.794823 16998 raft_consensus.cc:3037] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:49.794878 16888 raft_consensus.cc:2454] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cef743a355e04298af74ac72020e58d7 in term 0.
I20260810 04:45:49.795851 16998 raft_consensus.cc:504] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.796036 16998 leader_election.cc:288] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [CANDIDATE]: Term 1 election: Requested vote from peers 8d07e0e6dedb4fd3a5e062c3c54503f3 (127.14.237.3:32969), c00e800155f14a308990508ad4e7d6ee (127.14.237.2:37693)
I20260810 04:45:49.796263 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e9e8d52a54a87b0e86623ef259791" candidate_uuid: "cef743a355e04298af74ac72020e58d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3"
I20260810 04:45:49.796281 16642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e9e8d52a54a87b0e86623ef259791" candidate_uuid: "cef743a355e04298af74ac72020e58d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c00e800155f14a308990508ad4e7d6ee"
I20260810 04:45:49.796355 16642 raft_consensus.cc:3037] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:49.796355 16888 raft_consensus.cc:3037] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:49.797241 16642 raft_consensus.cc:2454] T 4a0e9e8d52a54a87b0e86623ef259791 P c00e800155f14a308990508ad4e7d6ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cef743a355e04298af74ac72020e58d7 in term 1.
I20260810 04:45:49.797219 16888 raft_consensus.cc:2454] T 4a0e9e8d52a54a87b0e86623ef259791 P 8d07e0e6dedb4fd3a5e062c3c54503f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cef743a355e04298af74ac72020e58d7 in term 1.
I20260810 04:45:49.797390 16326 leader_election.cc:302] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c00e800155f14a308990508ad4e7d6ee, cef743a355e04298af74ac72020e58d7; no voters:
I20260810 04:45:49.797497 16998 raft_consensus.cc:2781] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:49.797681 16998 raft_consensus.cc:686] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [term 1 LEADER]: Becoming Leader. State: Replica: cef743a355e04298af74ac72020e58d7, State: Running, Role: LEADER
I20260810 04:45:49.797801 16998 consensus_queue.cc:237] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } }
I20260810 04:45:49.798626 16209 catalog_manager.cc:5495] T 4a0e9e8d52a54a87b0e86623ef259791 P cef743a355e04298af74ac72020e58d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cef743a355e04298af74ac72020e58d7 (127.14.237.1). New cstate: current_term: 1 leader_uuid: "cef743a355e04298af74ac72020e58d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef743a355e04298af74ac72020e58d7" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d07e0e6dedb4fd3a5e062c3c54503f3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c00e800155f14a308990508ad4e7d6ee" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37693 } health_report { overall_health: UNKNOWN } } }
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:45:49.846604 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.838602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.14.237.1:46573)
0810 04:45:49.838677 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.838680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.838802 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:45:49.838896 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.838897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.838911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.838982 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:49.839615 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.840175 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.840922 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.841124 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.841204 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.843792 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:49.843805 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:49.843806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:49.843820 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:49.845306 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.845682 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.845686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.845686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.845753 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.845979 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.845980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.845980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.846185 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:49.846376 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.846414 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.846489 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:45:49.869840 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.863100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.14.237.1:46573)
0810 04:45:49.863188 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.863190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.863237 (+ 47us) server_negotiation.cc:407] Connection header received
0810 04:45:49.863282 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.863283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.863293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.863343 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:49.863888 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.864351 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.865031 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.865208 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.865347 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.867493 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:49.867494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:49.867495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:49.867501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:49.868704 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.869021 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.869022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.869022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.869074 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.869326 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.869327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.869328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.869421 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:49.869498 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.869640 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.869714 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[16137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:45:49.892573 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:49.885863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.14.237.1:46573)
0810 04:45:49.885929 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:49.885931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:49.885973 (+ 42us) server_negotiation.cc:407] Connection header received
0810 04:45:49.886034 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:49.886035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:49.886044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:49.886092 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:49.886680 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.887105 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.887807 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:49.887970 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:49.888037 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:49.890382 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:49.890383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:49.890383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:49.890389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:49.891609 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.891938 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.891939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.891940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.891988 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:49.892166 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:49.892166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:49.892167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:49.892256 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:49.892343 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:49.892431 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 04:45:49.892485 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 04:45:49.893281 16365 tablet_service.cc:2908] Rejecting scan request for tablet 4a0e9e8d52a54a87b0e86623ef259791: Uninitialized: clean time has not yet been initialized
I20260810 04:45:49.894245 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16301
I20260810 04:45:49.910322 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16505
I20260810 04:45:49.916221 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16725
I20260810 04:45:49.924572 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16154
2026-08-10T04:45:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2447 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[17034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[17034](info): set up 2 sockets
Aug 10 04:45:49 dist-test-slave-8hhm krb5kdc[17034](info): commencing operation
krb5kdc: starting...
W20260810 04:45:51.182500 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:45:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:45:51.306253 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37355
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:37355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:51.389408 17059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:51.389601 17059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:51.389621 17059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:51.390718 17059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:45:51.390800 17059 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:45:51.390822 17059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:51.390861 17059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:51.390888 17059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:45:51.390903 17059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:45:51.392386 17059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:37355
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:51.392637 17059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:51.392863 17059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:51.397770 17072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:51.397822 17071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:51.397830 17075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:51.398355 17059 server_base.cc:1034] running on GCE node
I20260810 04:45:51.398627 17059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:51.398944 17059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:51.400149 17059 hybrid_clock.cc:648] HybridClock initialized: now 1786337151400139 us; error 39 us; skew 500 ppm
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:51.402927 17059 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:45:51.403357 17059 webserver.cc:460] Webserver started at http://127.14.237.62:44485/ using document root <none> and password file <none>
I20260810 04:45:51.403589 17059 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:51.403661 17059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:51.403813 17059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:51.404924 17059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "54506b29a58142598e49174a7372a8aa"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "0d6438ffb9ebccd16e455ae755c24447"
server_key_iv: "94c6a0a18d32b0798c6a6fefbc15ace1"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.405371 17059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "54506b29a58142598e49174a7372a8aa"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "0d6438ffb9ebccd16e455ae755c24447"
server_key_iv: "94c6a0a18d32b0798c6a6fefbc15ace1"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.406759 17059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:51.407492 17087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:51.407670 17059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:45:51.407770 17059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "54506b29a58142598e49174a7372a8aa"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "0d6438ffb9ebccd16e455ae755c24447"
server_key_iv: "94c6a0a18d32b0798c6a6fefbc15ace1"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.407861 17059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:51.431313 17059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:51.433830 17059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:51.434006 17059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:51.438705 17059 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:37355
I20260810 04:45:51.438817 17186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:37355 every 8 connection(s)
I20260810 04:45:51.439121 17059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:45:51.439862 17187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:51.441699 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 17059
I20260810 04:45:51.441807 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:45:51.441977 15284 external_mini_cluster.cc:1428] Setting key 274e12d593c1e6fb446f70cd7fe86e6d
I20260810 04:45:51.442692 17187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa: Bootstrap starting.
I20260810 04:45:51.443298 17187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:51.443660 17187 log.cc:824] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:51.444427 17187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa: No bootstrap required, opened a new log
I20260810 04:45:51.446341 17187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54506b29a58142598e49174a7372a8aa" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37355 } }
I20260810 04:45:51.446511 17187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:51.446560 17187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54506b29a58142598e49174a7372a8aa, State: Initialized, Role: FOLLOWER
I20260810 04:45:51.446705 17187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54506b29a58142598e49174a7372a8aa" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37355 } }
I20260810 04:45:51.446827 17187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:45:51.446863 17187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:45:51.446918 17187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:51.447467 17187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54506b29a58142598e49174a7372a8aa" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37355 } }
I20260810 04:45:51.447556 17187 leader_election.cc:302] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54506b29a58142598e49174a7372a8aa; no voters:
I20260810 04:45:51.447743 17187 leader_election.cc:288] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:45:51.447808 17196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:51.447986 17196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [term 1 LEADER]: Becoming Leader. State: Replica: 54506b29a58142598e49174a7372a8aa, State: Running, Role: LEADER
I20260810 04:45:51.448027 17187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:45:51.448101 17196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54506b29a58142598e49174a7372a8aa" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37355 } }
I20260810 04:45:51.448415 17196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54506b29a58142598e49174a7372a8aa. Latest consensus state: current_term: 1 leader_uuid: "54506b29a58142598e49174a7372a8aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54506b29a58142598e49174a7372a8aa" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37355 } } }
I20260810 04:45:51.448500 17196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [sys.catalog]: This master's current role is: LEADER
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:51.448431 17198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54506b29a58142598e49174a7372a8aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54506b29a58142598e49174a7372a8aa" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37355 } } }
I20260810 04:45:51.448678 17198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa [sys.catalog]: This master's current role is: LEADER
I20260810 04:45:51.449373 17211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:45:51.450366 17211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:45:51.451828 17211 catalog_manager.cc:1347] Generated new cluster ID: 89a408d8f17241b08e3d1746777ede87
I20260810 04:45:51.451879 17211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:45:51.452914 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.442506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.14.237.62:37355)
0810 04:45:51.442682 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.442691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.442723 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:45:51.443196 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.443204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.443251 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.443388 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:51.444161 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.445108 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.445886 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.446294 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.446419 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.449139 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:51.449144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:51.449144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:51.449159 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:51.451254 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.451667 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.451670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.451670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.451774 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.452002 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.452004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.452004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.452306 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:51.452446 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.452601 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.452731 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260810 04:45:51.462749 17211 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:45:51.463241 17211 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:45:51.470239 17211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa: Generated new TSK 0
I20260810 04:45:51.470440 17211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:45:51.503609 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37355
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:51.587661 17241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:51.587883 17241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:51.587924 17241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:51.589059 17241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:51.589108 17241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:51.589128 17241 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:45:51.589188 17241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:45:51.590613 17241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:51.590804 17241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:51.590970 17241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:51.596194 17250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:51.596172 17248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:51.596161 17254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:51.596324 17241 server_base.cc:1034] running on GCE node
I20260810 04:45:51.596624 17241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:51.596896 17241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:51.598060 17241 hybrid_clock.cc:648] HybridClock initialized: now 1786337151598057 us; error 38 us; skew 500 ppm
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:51.600575 17241 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:45:51.601011 17241 webserver.cc:460] Webserver started at http://127.14.237.1:46755/ using document root <none> and password file <none>
I20260810 04:45:51.601307 17241 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:51.601380 17241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:51.601543 17241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:51.602574 17241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "0bb7521921d74a538a1db8565a3ae339"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "bfd56588e4066cc566498fd3466cee8b"
server_key_iv: "11c77cef8c594a4db9a0c0570bc1ada8"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.602993 17241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "0bb7521921d74a538a1db8565a3ae339"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "bfd56588e4066cc566498fd3466cee8b"
server_key_iv: "11c77cef8c594a4db9a0c0570bc1ada8"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.604384 17241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:45:51.605141 17265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:51.605378 17241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:51.605487 17241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "0bb7521921d74a538a1db8565a3ae339"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "bfd56588e4066cc566498fd3466cee8b"
server_key_iv: "11c77cef8c594a4db9a0c0570bc1ada8"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.605564 17241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:51.612880 17241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:51.615273 17241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:51.615429 17241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:51.615921 17241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:51.615993 17241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:51.616029 17241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:51.616093 17241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:51.622409 17241 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:33955
I20260810 04:45:51.622498 17460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:33955 every 8 connection(s)
I20260810 04:45:51.622818 17241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:51.626144 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.616355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:53487 (local address 127.14.237.62:37355)
0810 04:45:51.616398 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.616400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.616620 (+ 220us) server_negotiation.cc:407] Connection header received
0810 04:45:51.617120 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.617122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.617136 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.617186 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:51.618020 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.618890 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.620445 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.620780 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.620863 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.623498 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:51.623499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:51.623502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:51.623513 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:51.624965 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.625387 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.625388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.625388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.625445 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.625643 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.625643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.625644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.625735 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:51.625780 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.625984 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.626045 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:45:51.626287 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.616359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37355 (local address 127.14.237.1:53487)
0810 04:45:51.616506 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.616538 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.617010 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.617283 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.617287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.617641 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:51.617904 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.617910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.618997 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.618998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.620342 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.620345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.620877 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.620878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.620973 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.621537 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:51.621548 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:51.623402 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:51.625042 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.625047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.625049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.625320 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.625504 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.625504 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.625504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.625601 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.625786 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:51.625791 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:51.625894 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.625966 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.626076 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260810 04:45:51.626739 17461 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37355
I20260810 04:45:51.626824 17461 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:51.627003 17461 heartbeater.cc:511] Master 127.14.237.62:37355 requested a full tablet report, sending...
I20260810 04:45:51.627549 17114 ts_manager.cc:194] Registered new tserver with Master: 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955)
I20260810 04:45:51.628264 17114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:53487
I20260810 04:45:51.629026 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 17241
I20260810 04:45:51.629132 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:45:51.629336 15284 external_mini_cluster.cc:1428] Setting key 95ff4fa2ce2c46ef4c63a5f96c46c4a1
I20260810 04:45:51.633805 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.627782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37355 (local address 127.14.237.1:53997)
0810 04:45:51.627905 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.627922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.627996 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.628124 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.628125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.628304 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:51.628383 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.628385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.629205 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.629207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.629826 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.629828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.630202 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.630203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.630277 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.630709 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:51.630713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:51.630978 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:51.632758 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.632758 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.632759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.632924 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.633142 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.633142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.633142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.633196 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.633549 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:51.633550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:51.633612 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.633668 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.633742 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260810 04:45:51.633935 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.627784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:53997 (local address 127.14.237.62:37355)
0810 04:45:51.627840 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.627841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.627981 (+ 140us) server_negotiation.cc:407] Connection header received
0810 04:45:51.628054 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.628055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.628065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.628119 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:51.628446 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.629124 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.629928 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.630129 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.630203 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.631088 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:51.631090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:51.631090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:51.631097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:51.632660 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.633011 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.633012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.633012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.633073 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.633339 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.633340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.633341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.633482 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:51.633551 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.633719 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.633805 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:45:51.656639 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37355
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:51.738242 17479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:51.738457 17479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:51.738480 17479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:51.739521 17479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:51.739559 17479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:51.739578 17479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:45:51.739595 17479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:45:51.740983 17479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:51.741264 17479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:51.741516 17479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:51.746474 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:51.746474 17486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:51.746562 17485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:51.746614 17479 server_base.cc:1034] running on GCE node
I20260810 04:45:51.746924 17479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:51.747189 17479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:51.748323 17479 hybrid_clock.cc:648] HybridClock initialized: now 1786337151748302 us; error 34 us; skew 500 ppm
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:51.751026 17479 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:45:51.751487 17479 webserver.cc:460] Webserver started at http://127.14.237.2:41905/ using document root <none> and password file <none>
I20260810 04:45:51.751730 17479 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:51.751780 17479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:51.751919 17479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:51.752942 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "42d31025ece94238b6109c698390f83e"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "cff3040e01333b2be5f49e8472d57e74"
server_key_iv: "857fe41b1560e7985d698b1eb05c6a41"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.753358 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "42d31025ece94238b6109c698390f83e"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "cff3040e01333b2be5f49e8472d57e74"
server_key_iv: "857fe41b1560e7985d698b1eb05c6a41"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.755009 17479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:51.755810 17501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:51.755995 17479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:51.756099 17479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "42d31025ece94238b6109c698390f83e"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "cff3040e01333b2be5f49e8472d57e74"
server_key_iv: "857fe41b1560e7985d698b1eb05c6a41"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.756201 17479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:51.770165 17479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:51.772173 17479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:51.772334 17479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:51.772910 17479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:51.772966 17479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:51.773001 17479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:51.773020 17479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:51.780570 17479 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:36049
I20260810 04:45:51.780637 17696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:36049 every 8 connection(s)
I20260810 04:45:51.781008 17479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:45:51.781662 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 17479
I20260810 04:45:51.781762 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:45:51.781944 15284 external_mini_cluster.cc:1428] Setting key e5d92e242b191101cfdeb4ae58ff545e
I20260810 04:45:51.783186 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.773317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:56507 (local address 127.14.237.62:37355)
0810 04:45:51.773396 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.773398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.773715 (+ 317us) server_negotiation.cc:407] Connection header received
0810 04:45:51.774425 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.774426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.774456 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.774520 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:51.775412 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.776035 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.777702 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.777905 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.777992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.780098 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:51.780100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:51.780100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:51.780108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:51.781718 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.782211 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.782213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.782213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.782282 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.782540 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.782541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.782541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.782667 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:51.782748 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.783012 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.783094 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:45:51.783380 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.773332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37355 (local address 127.14.237.2:56507)
0810 04:45:51.773545 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.773598 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.774318 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.774626 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.774630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.775032 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:51.775323 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.775331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.776109 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.776110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.777595 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.777598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.778014 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.778015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.778147 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.778662 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:51.778676 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:51.779999 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:51.781833 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.781839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.781841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.782109 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.782352 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.782353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.782353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.782468 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.782738 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:51.782744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:51.782882 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.782990 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.783151 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260810 04:45:51.783931 17699 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37355
I20260810 04:45:51.784027 17699 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:51.784210 17699 heartbeater.cc:511] Master 127.14.237.62:37355 requested a full tablet report, sending...
I20260810 04:45:51.784605 17115 ts_manager.cc:194] Registered new tserver with Master: 42d31025ece94238b6109c698390f83e (127.14.237.2:36049)
I20260810 04:45:51.785138 17115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:56507
I20260810 04:45:51.790406 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.785056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:35433 (local address 127.14.237.62:37355)
0810 04:45:51.785091 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.785093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.785248 (+ 155us) server_negotiation.cc:407] Connection header received
0810 04:45:51.785364 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.785364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.785373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.785435 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:51.785772 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.786218 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.786890 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.787087 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.787173 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.788027 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:51.788028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:51.788029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:51.788035 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:51.789364 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.789686 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.789687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.789688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.789737 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.789931 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.789931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.789932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.790063 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:51.790142 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.790263 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.790325 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:45:51.790418 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.784982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37355 (local address 127.14.237.2:35433)
0810 04:45:51.785146 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.785156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.785247 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.785443 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.785444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.785638 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:51.785689 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.785690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.786314 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.786315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.786804 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.786806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.787177 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.787178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.787247 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.787662 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:51.787666 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:51.787924 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:51.789453 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.789454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.789454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.789612 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.789809 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.789809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.789810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.789864 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.790128 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:51.790129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:51.790171 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.790235 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.790326 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:45:51.809618 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37355
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:51.894244 17713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:51.894438 17713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:51.894462 17713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:51.895577 17713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:51.895620 17713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:51.895640 17713 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:45:51.895658 17713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:45:51.897099 17713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:37355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:51.897378 17713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:51.897559 17713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:51.902514 17726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:51.902489 17723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:51.902505 17724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:51.902881 17713 server_base.cc:1034] running on GCE node
I20260810 04:45:51.903102 17713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:51.903312 17713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:51.904448 17713 hybrid_clock.cc:648] HybridClock initialized: now 1786337151904423 us; error 46 us; skew 500 ppm
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:51.907043 17713 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:45:51.907548 17713 webserver.cc:460] Webserver started at http://127.14.237.3:34241/ using document root <none> and password file <none>
I20260810 04:45:51.907750 17713 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:51.907790 17713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:51.907876 17713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:51.908905 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "a39ffe7ee98c45b48760c34c157f77cb"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "baf1d3c53405777d99b5b7c731decb2b"
server_key_iv: "a0d89f14089106411aa11cc77e7c3141"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.909204 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "a39ffe7ee98c45b48760c34c157f77cb"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "baf1d3c53405777d99b5b7c731decb2b"
server_key_iv: "a0d89f14089106411aa11cc77e7c3141"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.910568 17713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:51.911391 17736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:51.911615 17713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:51.911690 17713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "a39ffe7ee98c45b48760c34c157f77cb"
format_stamp: "Formatted at 2026-08-10 04:45:51 on dist-test-slave-8hhm"
server_key: "baf1d3c53405777d99b5b7c731decb2b"
server_key_iv: "a0d89f14089106411aa11cc77e7c3141"
server_key_version: "encryptionkey@0"
I20260810 04:45:51.911789 17713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:51.919615 17713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:51.921437 17713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:51.921586 17713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:51.922106 17713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:51.922147 17713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:51.922168 17713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:51.922200 17713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:51.929368 17713 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:45277
I20260810 04:45:51.929473 17946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:45277 every 8 connection(s)
I20260810 04:45:51.929828 17713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:45:51.931888 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.922485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:47147 (local address 127.14.237.62:37355)
0810 04:45:51.922546 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.922548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.922732 (+ 184us) server_negotiation.cc:407] Connection header received
0810 04:45:51.923195 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.923197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.923212 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.923308 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:51.924061 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.924853 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.926550 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.926772 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.926830 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.929002 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:51.929003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:51.929004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:51.929011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:51.930688 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.931148 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.931149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.931149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.931204 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.931382 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.931383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.931383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.931476 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:51.931522 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.931754 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.931824 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:45:51.932030 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.922488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37355 (local address 127.14.237.3:47147)
0810 04:45:51.922626 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.922658 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.923097 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.923403 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.923407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.923766 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:51.923961 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.923966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.924956 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.924957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.926458 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.926462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.926854 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.926855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.927040 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.927568 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:51.927576 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:51.928905 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:51.930763 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.930767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.930769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.931072 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.931250 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.931250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.931251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.931347 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.931531 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:51.931536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:51.931639 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.931718 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.931848 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260810 04:45:51.932540 17947 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37355
I20260810 04:45:51.932624 17947 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:51.932818 17947 heartbeater.cc:511] Master 127.14.237.62:37355 requested a full tablet report, sending...
I20260810 04:45:51.933204 17114 ts_manager.cc:194] Registered new tserver with Master: a39ffe7ee98c45b48760c34c157f77cb (127.14.237.3:45277)
I20260810 04:45:51.933705 17114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:47147
I20260810 04:45:51.935101 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 17713
I20260810 04:45:51.935174 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:45:51.935370 15284 external_mini_cluster.cc:1428] Setting key 90dbf9ef1e2f5d57b39f9ded1bf4e101
I20260810 04:45:51.936251 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:45:51.939354 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.933602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37355 (local address 127.14.237.3:52477)
0810 04:45:51.933733 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.933744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.933810 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.933937 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.933938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.934132 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:51.934187 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.934188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.934893 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.934894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.935563 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.935565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.935969 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.935970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.936041 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.936448 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:51.936453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:51.936807 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:51.938470 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.938471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.938471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.938625 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.938825 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:51.938826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:51.938826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:51.938877 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:51.939131 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:51.939131 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:51.939164 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.939212 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.939285 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260810 04:45:51.939420 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.933689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:52477 (local address 127.14.237.62:37355)
0810 04:45:51.933728 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.933730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.933852 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:45:51.933874 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.933874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.933885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.933931 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:51.934277 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.934783 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.935645 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.935885 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.935956 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.936893 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:51.936894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:51.936894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:51.936900 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:51.938375 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.938695 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.938696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.938697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.938759 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.938937 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.938938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.938939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.939070 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:51.939138 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.939241 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.939325 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:45:51 dist-test-slave-8hhm krb5kdc[17034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:51.960996 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.954036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.14.237.62:37355)
0810 04:45:51.954095 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.954097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.954106 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:45:51.954131 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.954132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.954142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.954216 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:51.954761 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.955238 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.955963 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.956147 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.956265 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.958514 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:51.958516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:51.958516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:51.958524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:51.959813 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.960189 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.960190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.960191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.960257 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:51.960478 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:51.960479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:51.960480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:51.960585 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:51.960688 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.960818 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.960895 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:45:51.962692 17114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:45:51.964257 17114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:51.970502 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.966384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:36049 (local address 127.0.0.1:37050)
0810 04:45:51.966536 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.966575 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.966634 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.967014 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.967017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.967019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:51.967090 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.967091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.968456 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.968457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.969471 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.969473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.970196 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.970197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.970280 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.970295 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.970368 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.970420 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260810 04:45:51.970515 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.966477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.14.237.2:36049)
0810 04:45:51.966823 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.966828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.966840 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:45:51.966878 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.966880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.966896 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.966996 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:51.967235 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.968367 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.969550 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.970074 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.970165 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.970263 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.970383 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.970419 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":71,"threads_started":1}
I20260810 04:45:51.970830 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.966625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.14.237.1:33955)
0810 04:45:51.966810 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.966815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.967134 (+ 319us) server_negotiation.cc:407] Connection header received
0810 04:45:51.967182 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.967184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.967199 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.967272 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:51.967481 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.968524 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.969615 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.970350 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.970410 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.970461 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.970665 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.970733 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260810 04:45:51.970969 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.966620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.14.237.3:45277)
0810 04:45:51.966810 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.966815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.966869 (+ 54us) server_negotiation.cc:407] Connection header received
0810 04:45:51.966990 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.966992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.967008 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.967088 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:51.967273 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.968028 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.969441 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.970228 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.970319 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.970379 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.970799 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.970838 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260810 04:45:51.970754 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.966394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:33955 (local address 127.0.0.1:45164)
0810 04:45:51.966981 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.966990 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.967040 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.967347 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.967348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.967350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:51.967389 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.967390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.968610 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.968611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.969517 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.969519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.970491 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.970492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.970561 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.970564 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.970639 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.970671 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":44,"threads_started":1}
I20260810 04:45:51.971112 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.966386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:45277 (local address 127.0.0.1:45294)
0810 04:45:51.966809 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.966820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.966881 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.967156 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.967157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.967158 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:51.967197 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.967198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.968125 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.968126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.969363 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.969366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.970419 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.970420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.970488 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.970491 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.970540 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.970584 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":52,"threads_started":1}
I20260810 04:45:51.971665 17823 tablet_service.cc:1467] Processing CreateTablet for tablet d29d85d5c41c46deb9ac8a8715b94028 (DEFAULT_TABLE table=test-table [id=c75d14d93e3a44a398886f1c7b700c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:51.971662 17594 tablet_service.cc:1467] Processing CreateTablet for tablet d29d85d5c41c46deb9ac8a8715b94028 (DEFAULT_TABLE table=test-table [id=c75d14d93e3a44a398886f1c7b700c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:51.971756 17350 tablet_service.cc:1467] Processing CreateTablet for tablet d29d85d5c41c46deb9ac8a8715b94028 (DEFAULT_TABLE table=test-table [id=c75d14d93e3a44a398886f1c7b700c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:51.972062 17823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29d85d5c41c46deb9ac8a8715b94028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:51.972062 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29d85d5c41c46deb9ac8a8715b94028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:51.972141 17350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d29d85d5c41c46deb9ac8a8715b94028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:51.974984 17977 tablet_bootstrap.cc:492] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: Bootstrap starting.
I20260810 04:45:51.975366 17979 tablet_bootstrap.cc:492] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: Bootstrap starting.
I20260810 04:45:51.975723 17980 tablet_bootstrap.cc:492] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: Bootstrap starting.
I20260810 04:45:51.976148 17977 tablet_bootstrap.cc:654] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:51.976397 17980 tablet_bootstrap.cc:654] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:51.976399 17979 tablet_bootstrap.cc:654] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:51.976509 17977 log.cc:824] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:51.976745 17979 log.cc:824] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:51.976766 17980 log.cc:824] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:51.977394 17977 tablet_bootstrap.cc:492] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: No bootstrap required, opened a new log
I20260810 04:45:51.977475 17977 ts_tablet_manager.cc:1397] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:45:51.977998 17979 tablet_bootstrap.cc:492] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: No bootstrap required, opened a new log
I20260810 04:45:51.978071 17980 tablet_bootstrap.cc:492] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: No bootstrap required, opened a new log
I20260810 04:45:51.978084 17979 ts_tablet_manager.cc:1397] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:45:51.978140 17980 ts_tablet_manager.cc:1397] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:51.979539 17977 raft_consensus.cc:348] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.979719 17977 raft_consensus.cc:374] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:51.979760 17977 raft_consensus.cc:729] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb7521921d74a538a1db8565a3ae339, State: Initialized, Role: FOLLOWER
I20260810 04:45:51.979667 17979 raft_consensus.cc:348] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.979867 17979 raft_consensus.cc:374] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:51.979902 17979 raft_consensus.cc:729] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39ffe7ee98c45b48760c34c157f77cb, State: Initialized, Role: FOLLOWER
I20260810 04:45:51.979910 17977 consensus_queue.cc:260] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.980002 17979 consensus_queue.cc:260] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.980070 17980 raft_consensus.cc:348] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.980181 17977 ts_tablet_manager.cc:1428] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:45:51.980229 17980 raft_consensus.cc:374] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:51.980273 17980 raft_consensus.cc:729] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d31025ece94238b6109c698390f83e, State: Initialized, Role: FOLLOWER
I20260810 04:45:51.980298 17979 ts_tablet_manager.cc:1428] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:45:51.980319 17461 heartbeater.cc:503] Master 127.14.237.62:37355 was elected leader, sending a full tablet report...
I20260810 04:45:51.980445 17947 heartbeater.cc:503] Master 127.14.237.62:37355 was elected leader, sending a full tablet report...
I20260810 04:45:51.980459 17980 consensus_queue.cc:260] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.980623 17995 raft_consensus.cc:482] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:51.980736 17980 ts_tablet_manager.cc:1428] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:45:51.980752 17995 raft_consensus.cc:504] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.981003 17699 heartbeater.cc:503] Master 127.14.237.62:37355 was elected leader, sending a full tablet report...
I20260810 04:45:51.981178 17995 leader_election.cc:288] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a39ffe7ee98c45b48760c34c157f77cb (127.14.237.3:45277), 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955)
I20260810 04:45:51.984244 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.981462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:60327 (local address 127.14.237.1:33955)
0810 04:45:51.981500 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.981502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.981512 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:45:51.981550 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.981551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.981559 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.981614 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:51.981797 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.982399 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.983318 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.983844 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.983918 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.983927 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.984140 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.984187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:45:51.984231 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.981346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:33955 (local address 127.14.237.2:60327)
0810 04:45:51.981441 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.981453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.981492 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.981635 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.981636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.981638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:51.981703 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.981705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.982484 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.982485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.983201 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.983203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.983951 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.983952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.984056 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.984060 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.984117 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.984159 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 04:45:51.984306 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.981563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:59683 (local address 127.14.237.3:45277)
0810 04:45:51.981610 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.981612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.981665 (+ 53us) server_negotiation.cc:407] Connection header received
0810 04:45:51.981740 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.981741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.981750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.981796 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:51.981989 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.982618 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.983432 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.983985 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.984043 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.984048 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.984205 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.984245 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:45:51.984395 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.981351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:45277 (local address 127.14.237.2:59683)
0810 04:45:51.981583 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:51.981615 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:51.981690 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:51.981839 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:51.981839 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:51.981842 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:51.981903 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.981904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.982693 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.982693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.983360 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:51.983362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.984057 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.984057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:51.984138 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.984141 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:45:51.984191 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:45:51.984219 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":8,"thread_start_us":75,"threads_started":1}
I20260810 04:45:51.984555 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29d85d5c41c46deb9ac8a8715b94028" candidate_uuid: "42d31025ece94238b6109c698390f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7521921d74a538a1db8565a3ae339" is_pre_election: true
I20260810 04:45:51.984664 17372 raft_consensus.cc:2454] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42d31025ece94238b6109c698390f83e in term 0.
I20260810 04:45:51.984616 17859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29d85d5c41c46deb9ac8a8715b94028" candidate_uuid: "42d31025ece94238b6109c698390f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39ffe7ee98c45b48760c34c157f77cb" is_pre_election: true
I20260810 04:45:51.984743 17859 raft_consensus.cc:2454] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42d31025ece94238b6109c698390f83e in term 0.
I20260810 04:45:51.984882 17506 leader_election.cc:302] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb7521921d74a538a1db8565a3ae339, 42d31025ece94238b6109c698390f83e; no voters:
I20260810 04:45:51.985054 17996 raft_consensus.cc:2781] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:51.985131 17996 raft_consensus.cc:482] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:51.985153 17996 raft_consensus.cc:3037] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:51.986150 17996 raft_consensus.cc:504] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.986289 17996 leader_election.cc:288] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [CANDIDATE]: Term 1 election: Requested vote from peers a39ffe7ee98c45b48760c34c157f77cb (127.14.237.3:45277), 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955)
I20260810 04:45:51.986447 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29d85d5c41c46deb9ac8a8715b94028" candidate_uuid: "42d31025ece94238b6109c698390f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7521921d74a538a1db8565a3ae339"
I20260810 04:45:51.986449 17859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d29d85d5c41c46deb9ac8a8715b94028" candidate_uuid: "42d31025ece94238b6109c698390f83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39ffe7ee98c45b48760c34c157f77cb"
I20260810 04:45:51.986536 17372 raft_consensus.cc:3037] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:51.986550 17859 raft_consensus.cc:3037] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:51.987188 17372 raft_consensus.cc:2454] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42d31025ece94238b6109c698390f83e in term 1.
I20260810 04:45:51.987192 17859 raft_consensus.cc:2454] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42d31025ece94238b6109c698390f83e in term 1.
I20260810 04:45:51.987349 17506 leader_election.cc:302] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb7521921d74a538a1db8565a3ae339, 42d31025ece94238b6109c698390f83e; no voters:
I20260810 04:45:51.987478 17996 raft_consensus.cc:2781] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:51.987547 17996 raft_consensus.cc:686] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 1 LEADER]: Becoming Leader. State: Replica: 42d31025ece94238b6109c698390f83e, State: Running, Role: LEADER
I20260810 04:45:51.987622 17996 consensus_queue.cc:237] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } }
I20260810 04:45:51.988360 17114 catalog_manager.cc:5495] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e reported cstate change: term changed from 0 to 1, leader changed from <none> to 42d31025ece94238b6109c698390f83e (127.14.237.2). New cstate: current_term: 1 leader_uuid: "42d31025ece94238b6109c698390f83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } health_report { overall_health: UNKNOWN } } }
I20260810 04:45:51.996136 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:51.994018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.14.237.2:36049)
0810 04:45:51.994062 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:51.994064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:51.994088 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:45:51.994158 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:51.994158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:51.994173 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:51.994224 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:51.994367 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.994800 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.995511 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:51.995673 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:51.995720 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:51.995884 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:51.995963 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:51.996016 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:51.996058 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 04:45:51.996073 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:45:51.997726 17859 raft_consensus.cc:1261] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Refusing update from remote peer 42d31025ece94238b6109c698390f83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:51.997756 17372 raft_consensus.cc:1261] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Refusing update from remote peer 42d31025ece94238b6109c698390f83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:51.998001 17996 consensus_queue.cc:1035] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:51.998108 17995 consensus_queue.cc:1035] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:52.000545 18005 mvcc.cc:205] Tried to move back new op lower bound from 7316836974580228096 to 7316836974542036992. Current Snapshot: MvccSnapshot[applied={T|T < 7316836974580228096}]
I20260810 04:45:52.001157 18006 mvcc.cc:205] Tried to move back new op lower bound from 7316836974580228096 to 7316836974542036992. Current Snapshot: MvccSnapshot[applied={T|T < 7316836974580228096}]
I20260810 04:45:52.003760 17115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:52.003837 17115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:52.004609 17115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa: Sending DeleteTablet for 3 replicas of tablet d29d85d5c41c46deb9ac8a8715b94028
I20260810 04:45:52.004947 17594 tablet_service.cc:1514] Processing DeleteTablet for tablet d29d85d5c41c46deb9ac8a8715b94028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:51 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:37050
I20260810 04:45:52.005046 17823 tablet_service.cc:1514] Processing DeleteTablet for tablet d29d85d5c41c46deb9ac8a8715b94028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:51 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:45294
I20260810 04:45:52.005052 17350 tablet_service.cc:1514] Processing DeleteTablet for tablet d29d85d5c41c46deb9ac8a8715b94028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:51 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:45164
I20260810 04:45:52.005188 18023 tablet_replica.cc:330] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: stopping tablet replica
I20260810 04:45:52.005306 18028 tablet_replica.cc:330] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: stopping tablet replica
I20260810 04:45:52.005326 18023 raft_consensus.cc:2229] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:45:52.005399 18028 raft_consensus.cc:2229] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:52.005434 18023 raft_consensus.cc:2258] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:52.005578 17115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:45:52.005715 17115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:52.005757 18028 raft_consensus.cc:2258] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:52.005858 18023 ts_tablet_manager.cc:1905] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:52.006160 18028 ts_tablet_manager.cc:1905] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:52.007266 17823 tablet_service.cc:1467] Processing CreateTablet for tablet 544b969dedea49319414a6ce4c7d4dbf (DEFAULT_TABLE table=test-table [id=0e58b10d4b4e4e48b5cd7dbead18ea0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:52.007391 17823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544b969dedea49319414a6ce4c7d4dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:52.007658 18023 ts_tablet_manager.cc:1918] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:45:52.007737 18023 log.cc:1196] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/d29d85d5c41c46deb9ac8a8715b94028
I20260810 04:45:52.007769 18028 ts_tablet_manager.cc:1918] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:45:52.007838 18028 log.cc:1196] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/d29d85d5c41c46deb9ac8a8715b94028
I20260810 04:45:52.007987 18023 ts_tablet_manager.cc:1939] T d29d85d5c41c46deb9ac8a8715b94028 P 42d31025ece94238b6109c698390f83e: Deleting consensus metadata
I20260810 04:45:52.008105 18028 ts_tablet_manager.cc:1939] T d29d85d5c41c46deb9ac8a8715b94028 P 0bb7521921d74a538a1db8565a3ae339: Deleting consensus metadata
I20260810 04:45:52.008991 17979 tablet_bootstrap.cc:492] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: Bootstrap starting.
I20260810 04:45:52.009088 17090 catalog_manager.cc:4841] TS 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955): tablet d29d85d5c41c46deb9ac8a8715b94028 (table test-table [id=c75d14d93e3a44a398886f1c7b700c67]) successfully deleted
I20260810 04:45:52.009342 17350 tablet_service.cc:1467] Processing CreateTablet for tablet 544b969dedea49319414a6ce4c7d4dbf (DEFAULT_TABLE table=test-table [id=0e58b10d4b4e4e48b5cd7dbead18ea0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:52.009454 17350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544b969dedea49319414a6ce4c7d4dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:52.009456 17979 tablet_bootstrap.cc:654] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:52.010144 17979 tablet_bootstrap.cc:492] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: No bootstrap required, opened a new log
I20260810 04:45:52.010187 17979 ts_tablet_manager.cc:1397] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:52.010331 17979 raft_consensus.cc:348] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.010389 17979 raft_consensus.cc:374] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:52.010409 17979 raft_consensus.cc:729] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39ffe7ee98c45b48760c34c157f77cb, State: Initialized, Role: FOLLOWER
I20260810 04:45:52.010454 17979 consensus_queue.cc:260] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.010490 17977 tablet_bootstrap.cc:492] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: Bootstrap starting.
I20260810 04:45:52.010541 17979 ts_tablet_manager.cc:1428] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:45:52.010601 17092 catalog_manager.cc:4841] TS 42d31025ece94238b6109c698390f83e (127.14.237.2:36049): tablet d29d85d5c41c46deb9ac8a8715b94028 (table test-table [id=c75d14d93e3a44a398886f1c7b700c67]) successfully deleted
I20260810 04:45:52.010864 17594 tablet_service.cc:1467] Processing CreateTablet for tablet 544b969dedea49319414a6ce4c7d4dbf (DEFAULT_TABLE table=test-table [id=0e58b10d4b4e4e48b5cd7dbead18ea0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:52.010910 17977 tablet_bootstrap.cc:654] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:52.010958 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544b969dedea49319414a6ce4c7d4dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:52.011982 17977 tablet_bootstrap.cc:492] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: No bootstrap required, opened a new log
I20260810 04:45:52.012023 17977 ts_tablet_manager.cc:1397] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:52.012172 17977 raft_consensus.cc:348] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.012265 17977 raft_consensus.cc:374] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:52.012290 17977 raft_consensus.cc:729] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb7521921d74a538a1db8565a3ae339, State: Initialized, Role: FOLLOWER
I20260810 04:45:52.012351 17980 tablet_bootstrap.cc:492] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: Bootstrap starting.
I20260810 04:45:52.012355 17977 consensus_queue.cc:260] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.012480 17977 ts_tablet_manager.cc:1428] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:52.012748 17980 tablet_bootstrap.cc:654] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:52.013414 17980 tablet_bootstrap.cc:492] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: No bootstrap required, opened a new log
I20260810 04:45:52.013466 17980 ts_tablet_manager.cc:1397] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:52.013619 17980 raft_consensus.cc:348] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.013707 17980 raft_consensus.cc:374] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:52.013736 17980 raft_consensus.cc:729] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d31025ece94238b6109c698390f83e, State: Initialized, Role: FOLLOWER
I20260810 04:45:52.013775 17980 consensus_queue.cc:260] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.013860 17980 ts_tablet_manager.cc:1428] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:52.017303 18027 tablet_replica.cc:330] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: stopping tablet replica
I20260810 04:45:52.017413 18027 raft_consensus.cc:2229] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:52.017539 18027 raft_consensus.cc:2258] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:52.017917 18027 ts_tablet_manager.cc:1905] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:52.019105 18027 ts_tablet_manager.cc:1918] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:45:52.019162 18027 log.cc:1196] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/d29d85d5c41c46deb9ac8a8715b94028
I20260810 04:45:52.019393 18027 ts_tablet_manager.cc:1939] T d29d85d5c41c46deb9ac8a8715b94028 P a39ffe7ee98c45b48760c34c157f77cb: Deleting consensus metadata
I20260810 04:45:52.019739 17089 catalog_manager.cc:4841] TS a39ffe7ee98c45b48760c34c157f77cb (127.14.237.3:45277): tablet d29d85d5c41c46deb9ac8a8715b94028 (table test-table [id=c75d14d93e3a44a398886f1c7b700c67]) successfully deleted
W20260810 04:45:52.031644 17701 tablet.cc:2369] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:45:52.123674 17462 tablet.cc:2369] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:45:52.180469 17948 tablet.cc:2369] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:45:52.271615 17992 raft_consensus.cc:482] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:52.271726 17992 raft_consensus.cc:504] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.272080 17992 leader_election.cc:288] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955), 42d31025ece94238b6109c698390f83e (127.14.237.2:36049)
I20260810 04:45:52.276237 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.272393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:40451 (local address 127.14.237.2:36049)
0810 04:45:52.272510 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:52.272512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:52.272596 (+ 84us) server_negotiation.cc:407] Connection header received
0810 04:45:52.272656 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:52.272657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:52.272670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:52.272713 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:52.272916 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.273722 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.275292 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.275858 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.275939 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.275948 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:52.276125 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 04:45:52.276169 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 04:45:52.276237 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.272346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:37683 (local address 127.14.237.1:33955)
0810 04:45:52.272465 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:52.272468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:52.272748 (+ 280us) server_negotiation.cc:407] Connection header received
0810 04:45:52.272772 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:52.272773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:52.272780 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:52.272820 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:52.273007 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.273744 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.275291 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.275857 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.275939 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.275948 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:52.276127 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 04:45:52.276165 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 04:45:52.276279 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.272274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:36049 (local address 127.14.237.3:40451)
0810 04:45:52.272345 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:52.272532 (+ 187us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:52.272593 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:52.272750 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:52.272751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:52.272752 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:52.272841 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.272843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.273828 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.273829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.275182 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.275186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.275957 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.275957 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.276039 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.276041 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:45:52.276097 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:45:52.276154 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":104}
I20260810 04:45:52.276295 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.272275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:33955 (local address 127.14.237.3:37683)
0810 04:45:52.272512 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:52.272522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:52.272593 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:52.272842 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:52.272843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:52.272845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:52.272920 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.272922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.273828 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.273829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.275182 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.275186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.275957 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.275958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.276034 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.276038 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:45:52.276097 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:45:52.276153 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":143,"thread_start_us":143,"threads_started":1}
I20260810 04:45:52.276585 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544b969dedea49319414a6ce4c7d4dbf" candidate_uuid: "a39ffe7ee98c45b48760c34c157f77cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d31025ece94238b6109c698390f83e" is_pre_election: true
I20260810 04:45:52.276746 17625 raft_consensus.cc:2454] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39ffe7ee98c45b48760c34c157f77cb in term 0.
I20260810 04:45:52.276714 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544b969dedea49319414a6ce4c7d4dbf" candidate_uuid: "a39ffe7ee98c45b48760c34c157f77cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7521921d74a538a1db8565a3ae339" is_pre_election: true
I20260810 04:45:52.276798 17372 raft_consensus.cc:2454] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39ffe7ee98c45b48760c34c157f77cb in term 0.
I20260810 04:45:52.276973 17743 leader_election.cc:302] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d31025ece94238b6109c698390f83e, a39ffe7ee98c45b48760c34c157f77cb; no voters:
I20260810 04:45:52.277145 17992 raft_consensus.cc:2781] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:52.277195 17992 raft_consensus.cc:482] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:52.277227 17992 raft_consensus.cc:3037] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:52.277944 17992 raft_consensus.cc:504] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.278093 17992 leader_election.cc:288] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [CANDIDATE]: Term 1 election: Requested vote from peers 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955), 42d31025ece94238b6109c698390f83e (127.14.237.2:36049)
I20260810 04:45:52.278295 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544b969dedea49319414a6ce4c7d4dbf" candidate_uuid: "a39ffe7ee98c45b48760c34c157f77cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7521921d74a538a1db8565a3ae339"
I20260810 04:45:52.278378 17372 raft_consensus.cc:3037] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:52.278328 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544b969dedea49319414a6ce4c7d4dbf" candidate_uuid: "a39ffe7ee98c45b48760c34c157f77cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d31025ece94238b6109c698390f83e"
I20260810 04:45:52.278412 17625 raft_consensus.cc:3037] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:52.278992 17372 raft_consensus.cc:2454] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39ffe7ee98c45b48760c34c157f77cb in term 1.
I20260810 04:45:52.279096 17625 raft_consensus.cc:2454] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39ffe7ee98c45b48760c34c157f77cb in term 1.
I20260810 04:45:52.279163 17742 leader_election.cc:302] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb7521921d74a538a1db8565a3ae339, a39ffe7ee98c45b48760c34c157f77cb; no voters:
I20260810 04:45:52.279259 17992 raft_consensus.cc:2781] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:52.279366 17992 raft_consensus.cc:686] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 1 LEADER]: Becoming Leader. State: Replica: a39ffe7ee98c45b48760c34c157f77cb, State: Running, Role: LEADER
I20260810 04:45:52.279435 17992 consensus_queue.cc:237] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.280169 17115 catalog_manager.cc:5495] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb reported cstate change: term changed from 0 to 1, leader changed from <none> to a39ffe7ee98c45b48760c34c157f77cb (127.14.237.3). New cstate: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } health_report { overall_health: HEALTHY } } }
Aug 10 04:45:52 dist-test-slave-8hhm krb5kdc[17034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:52.316705 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.309950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.14.237.62:37355)
0810 04:45:52.310010 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:52.310012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:52.310149 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:45:52.310191 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:52.310192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:52.310201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:52.310252 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:52.310715 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.311292 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.312006 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.312199 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.312251 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.314280 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:52.314281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:52.314281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:52.314288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:52.315746 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:52.316068 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:52.316068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:52.316069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:52.316137 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:52.316311 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:52.316312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:52.316312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:52.316406 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:52.316454 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:52.316565 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 04:45:52.316640 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:45:52.316843 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.309881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37355 (local address 127.0.0.1:40354)
0810 04:45:52.310076 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:52.310085 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:52.310148 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:52.310268 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:52.310269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:52.310585 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:52.310631 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.310633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.311359 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.311360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.311939 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.311941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.312265 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.312266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.312305 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.312793 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:52.312798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:52.314196 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:52.315812 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:52.315819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:52.315820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:52.316003 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:52.316194 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:52.316194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:52.316194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:52.316259 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:52.316458 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:52.316464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:52.316502 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 04:45:52.316550 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:45:52.316711 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":74,"thread_start_us":95,"threads_started":1}
I20260810 04:45:52.318969 17115 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:40354:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:45:52.319216 17115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:52.320749 17350 tablet_service.cc:1467] Processing CreateTablet for tablet 90c10f27ccc74b0c9c17914b3b455951 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb856fa9b23042d09502a1f6c3ad4bfd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:45:52.320780 17594 tablet_service.cc:1467] Processing CreateTablet for tablet 90c10f27ccc74b0c9c17914b3b455951 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb856fa9b23042d09502a1f6c3ad4bfd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:45:52.320842 17823 tablet_service.cc:1467] Processing CreateTablet for tablet 90c10f27ccc74b0c9c17914b3b455951 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb856fa9b23042d09502a1f6c3ad4bfd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:45:52.320945 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c10f27ccc74b0c9c17914b3b455951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:52.320945 17350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c10f27ccc74b0c9c17914b3b455951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:52.320964 17823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90c10f27ccc74b0c9c17914b3b455951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:52.322229 17977 tablet_bootstrap.cc:492] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: Bootstrap starting.
I20260810 04:45:52.322511 17980 tablet_bootstrap.cc:492] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: Bootstrap starting.
I20260810 04:45:52.322677 17977 tablet_bootstrap.cc:654] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:52.322912 17980 tablet_bootstrap.cc:654] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:52.323159 17979 tablet_bootstrap.cc:492] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: Bootstrap starting.
I20260810 04:45:52.323582 17977 tablet_bootstrap.cc:492] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: No bootstrap required, opened a new log
I20260810 04:45:52.323598 17979 tablet_bootstrap.cc:654] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:52.323642 17977 ts_tablet_manager.cc:1397] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:52.323813 17977 raft_consensus.cc:348] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.323901 17977 raft_consensus.cc:374] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:52.323926 17977 raft_consensus.cc:729] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb7521921d74a538a1db8565a3ae339, State: Initialized, Role: FOLLOWER
I20260810 04:45:52.324008 17977 consensus_queue.cc:260] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.324150 17977 ts_tablet_manager.cc:1428] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:52.324323 17980 tablet_bootstrap.cc:492] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: No bootstrap required, opened a new log
I20260810 04:45:52.324378 17980 ts_tablet_manager.cc:1397] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:52.324324 17991 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.324415 17991 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:52.324504 17980 raft_consensus.cc:348] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.324585 17980 raft_consensus.cc:374] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:52.324611 17980 raft_consensus.cc:729] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d31025ece94238b6109c698390f83e, State: Initialized, Role: FOLLOWER
I20260810 04:45:52.324663 17980 consensus_queue.cc:260] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.324775 17980 ts_tablet_manager.cc:1428] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:52.324793 17995 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.324889 17995 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:52.324981 17979 tablet_bootstrap.cc:492] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: No bootstrap required, opened a new log
I20260810 04:45:52.325047 17979 ts_tablet_manager.cc:1397] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:45:52.325383 17979 raft_consensus.cc:348] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.325448 17979 raft_consensus.cc:374] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:52.325461 17979 raft_consensus.cc:729] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39ffe7ee98c45b48760c34c157f77cb, State: Initialized, Role: FOLLOWER
I20260810 04:45:52.325521 17979 consensus_queue.cc:260] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.325616 17979 ts_tablet_manager.cc:1428] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:45:52.325677 17992 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.325771 17992 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:52.338774 17992 raft_consensus.cc:482] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:52.338853 17992 raft_consensus.cc:504] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.339016 17992 leader_election.cc:288] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955), 42d31025ece94238b6109c698390f83e (127.14.237.2:36049)
I20260810 04:45:52.339223 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c10f27ccc74b0c9c17914b3b455951" candidate_uuid: "a39ffe7ee98c45b48760c34c157f77cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7521921d74a538a1db8565a3ae339" is_pre_election: true
I20260810 04:45:52.339231 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c10f27ccc74b0c9c17914b3b455951" candidate_uuid: "a39ffe7ee98c45b48760c34c157f77cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d31025ece94238b6109c698390f83e" is_pre_election: true
I20260810 04:45:52.339313 17625 raft_consensus.cc:2454] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39ffe7ee98c45b48760c34c157f77cb in term 0.
I20260810 04:45:52.339313 17372 raft_consensus.cc:2454] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39ffe7ee98c45b48760c34c157f77cb in term 0.
I20260810 04:45:52.339442 17743 leader_election.cc:302] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d31025ece94238b6109c698390f83e, a39ffe7ee98c45b48760c34c157f77cb; no voters:
I20260810 04:45:52.339546 17992 raft_consensus.cc:2781] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:52.339586 17992 raft_consensus.cc:482] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:52.339602 17992 raft_consensus.cc:3037] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:52.340289 17992 raft_consensus.cc:504] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.340430 17992 leader_election.cc:288] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [CANDIDATE]: Term 1 election: Requested vote from peers 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955), 42d31025ece94238b6109c698390f83e (127.14.237.2:36049)
I20260810 04:45:52.340557 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c10f27ccc74b0c9c17914b3b455951" candidate_uuid: "a39ffe7ee98c45b48760c34c157f77cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7521921d74a538a1db8565a3ae339"
I20260810 04:45:52.340572 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c10f27ccc74b0c9c17914b3b455951" candidate_uuid: "a39ffe7ee98c45b48760c34c157f77cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d31025ece94238b6109c698390f83e"
I20260810 04:45:52.340659 17372 raft_consensus.cc:3037] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:52.340657 17625 raft_consensus.cc:3037] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:52.341321 17372 raft_consensus.cc:2454] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39ffe7ee98c45b48760c34c157f77cb in term 1.
I20260810 04:45:52.341349 17625 raft_consensus.cc:2454] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39ffe7ee98c45b48760c34c157f77cb in term 1.
I20260810 04:45:52.341531 17742 leader_election.cc:302] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb7521921d74a538a1db8565a3ae339, a39ffe7ee98c45b48760c34c157f77cb; no voters:
I20260810 04:45:52.341637 17992 raft_consensus.cc:2781] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:52.341712 17992 raft_consensus.cc:686] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [term 1 LEADER]: Becoming Leader. State: Replica: a39ffe7ee98c45b48760c34c157f77cb, State: Running, Role: LEADER
I20260810 04:45:52.341768 17992 consensus_queue.cc:237] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } }
I20260810 04:45:52.342080 18039 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: TxnStatusTablet state changed. Reason: New leader a39ffe7ee98c45b48760c34c157f77cb. Latest consensus state: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.342170 18039 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:45:52.342357 18061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:45:52.342501 18061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:45:52.342392 17114 catalog_manager.cc:5495] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb reported cstate change: term changed from 0 to 1, leader changed from <none> to a39ffe7ee98c45b48760c34c157f77cb (127.14.237.3). New cstate: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } health_report { overall_health: HEALTHY } } }
I20260810 04:45:52.354416 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.351512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.14.237.3:45277)
0810 04:45:52.351553 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:52.351555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:52.351603 (+ 48us) server_negotiation.cc:407] Connection header received
0810 04:45:52.351691 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:52.351691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:52.351705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:52.351764 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:52.351908 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.352438 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.353615 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.353803 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.353873 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.354075 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:52.354202 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:52.354270 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:52.354346 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 04:45:52.354364 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:45:52.354477 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.351463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:45277 (local address 127.0.0.1:45300)
0810 04:45:52.351551 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:52.351562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:52.351616 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:52.351769 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:52.351770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:52.351771 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:45:52.351836 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.351838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.352515 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.352516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.353541 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.353544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.353896 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.353897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.353963 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.353981 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:45:52.354276 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:45:52.354285 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 04:45:52.354343 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:45:52.354393 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 04:45:52.355628 17372 raft_consensus.cc:1261] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Refusing update from remote peer a39ffe7ee98c45b48760c34c157f77cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:52.355701 17625 raft_consensus.cc:1261] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Refusing update from remote peer a39ffe7ee98c45b48760c34c157f77cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:52.355954 18039 consensus_queue.cc:1035] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:52.356140 18039 consensus_queue.cc:1035] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:52.356751 17991 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: TxnStatusTablet state changed. Reason: New leader a39ffe7ee98c45b48760c34c157f77cb. Latest consensus state: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.356849 17991 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:52.356923 17995 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: TxnStatusTablet state changed. Reason: New leader a39ffe7ee98c45b48760c34c157f77cb. Latest consensus state: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.356999 17995 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:52.357461 17992 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.357549 17992 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:45:52.358341 17991 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.358422 17991 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P 0bb7521921d74a538a1db8565a3ae339: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:52.358765 17992 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.358855 17992 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P a39ffe7ee98c45b48760c34c157f77cb: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:45:52.359434 18004 mvcc.cc:205] Tried to move back new op lower bound from 7316836976046698496 to 7316836975992168448. Current Snapshot: MvccSnapshot[applied={T|T < 7316836976046698496}]
I20260810 04:45:52.359524 17995 tablet_replica.cc:439] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a39ffe7ee98c45b48760c34c157f77cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 } } peers { permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 } } peers { permanent_uuid: "a39ffe7ee98c45b48760c34c157f77cb" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45277 } } }
I20260810 04:45:52.359596 17995 tablet_replica.cc:442] T 90c10f27ccc74b0c9c17914b3b455951 P 42d31025ece94238b6109c698390f83e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:52.361941 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.359713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.14.237.3:45277)
0810 04:45:52.359746 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:52.359747 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:52.359787 (+ 40us) server_negotiation.cc:407] Connection header received
0810 04:45:52.359848 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:52.359849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:52.359858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:52.359903 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:52.360084 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.360523 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.361279 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.361482 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.361558 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.361716 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:52.361746 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:52.361792 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:52.361848 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 04:45:52.361865 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:45:52.367287 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.364740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:48641 (local address 127.14.237.3:45277)
0810 04:45:52.364797 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:52.364799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:52.364836 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:45:52.364854 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:52.364855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:52.364863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:52.364906 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:52.365052 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.365583 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.366656 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.366832 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.366884 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.367079 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:52.367107 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:52.367145 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:52.367224 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 04:45:52.367239 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:45:52.367377 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.364656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:45277 (local address 127.14.237.3:48641)
0810 04:45:52.364729 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:52.364739 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:52.364775 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:52.364931 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:52.364931 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:52.364933 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:45:52.364989 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.364991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.365664 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.365665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.366571 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:52.366573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.366906 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.366907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:52.366983 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.366999 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:45:52.367168 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:45:52.367169 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:45:52.367233 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:45:52.367276 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 04:45:52.369925 17372 raft_consensus.cc:1261] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Refusing update from remote peer a39ffe7ee98c45b48760c34c157f77cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:52.369930 17625 raft_consensus.cc:1261] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Refusing update from remote peer a39ffe7ee98c45b48760c34c157f77cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:52.370141 17992 consensus_queue.cc:1035] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb7521921d74a538a1db8565a3ae339" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:52.370330 18075 consensus_queue.cc:1035] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d31025ece94238b6109c698390f83e" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:52.385705 17115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:52.385830 17115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:52.386564 17115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54506b29a58142598e49174a7372a8aa: Sending DeleteTablet for 3 replicas of tablet 544b969dedea49319414a6ce4c7d4dbf
I20260810 04:45:52.386837 17821 tablet_service.cc:1514] Processing DeleteTablet for tablet 544b969dedea49319414a6ce4c7d4dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:52 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:45294
I20260810 04:45:52.386835 17350 tablet_service.cc:1514] Processing DeleteTablet for tablet 544b969dedea49319414a6ce4c7d4dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:52 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:45164
I20260810 04:45:52.386888 17594 tablet_service.cc:1514] Processing DeleteTablet for tablet 544b969dedea49319414a6ce4c7d4dbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:52 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:37050
I20260810 04:45:52.386937 18027 tablet_replica.cc:330] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: stopping tablet replica
I20260810 04:45:52.386938 18028 tablet_replica.cc:330] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: stopping tablet replica
I20260810 04:45:52.386986 18027 raft_consensus.cc:2229] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:45:52.386988 18028 raft_consensus.cc:2229] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:52.386981 18023 tablet_replica.cc:330] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: stopping tablet replica
I20260810 04:45:52.387024 18028 raft_consensus.cc:2258] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:52.387040 18023 raft_consensus.cc:2229] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:52.387068 18027 raft_consensus.cc:2258] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:52.387079 18023 raft_consensus.cc:2258] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:52.387383 18023 ts_tablet_manager.cc:1905] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:52.387476 18028 ts_tablet_manager.cc:1905] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:52.388032 18027 ts_tablet_manager.cc:1905] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:52.389158 18028 ts_tablet_manager.cc:1918] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:52.389180 18023 ts_tablet_manager.cc:1918] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:52.389247 18028 log.cc:1196] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/544b969dedea49319414a6ce4c7d4dbf
I20260810 04:45:52.389266 18023 log.cc:1196] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/544b969dedea49319414a6ce4c7d4dbf
I20260810 04:45:52.389489 18023 ts_tablet_manager.cc:1939] T 544b969dedea49319414a6ce4c7d4dbf P 42d31025ece94238b6109c698390f83e: Deleting consensus metadata
I20260810 04:45:52.389503 18028 ts_tablet_manager.cc:1939] T 544b969dedea49319414a6ce4c7d4dbf P 0bb7521921d74a538a1db8565a3ae339: Deleting consensus metadata
I20260810 04:45:52.389719 18027 ts_tablet_manager.cc:1918] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:52.389768 18027 log.cc:1196] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/544b969dedea49319414a6ce4c7d4dbf
I20260810 04:45:52.389786 17092 catalog_manager.cc:4841] TS 42d31025ece94238b6109c698390f83e (127.14.237.2:36049): tablet 544b969dedea49319414a6ce4c7d4dbf (table test-table [id=0e58b10d4b4e4e48b5cd7dbead18ea0e]) successfully deleted
I20260810 04:45:52.389827 17090 catalog_manager.cc:4841] TS 0bb7521921d74a538a1db8565a3ae339 (127.14.237.1:33955): tablet 544b969dedea49319414a6ce4c7d4dbf (table test-table [id=0e58b10d4b4e4e48b5cd7dbead18ea0e]) successfully deleted
I20260810 04:45:52.389981 18027 ts_tablet_manager.cc:1939] T 544b969dedea49319414a6ce4c7d4dbf P a39ffe7ee98c45b48760c34c157f77cb: Deleting consensus metadata
I20260810 04:45:52.390312 17089 catalog_manager.cc:4841] TS a39ffe7ee98c45b48760c34c157f77cb (127.14.237.3:45277): tablet 544b969dedea49319414a6ce4c7d4dbf (table test-table [id=0e58b10d4b4e4e48b5cd7dbead18ea0e]) successfully deleted
Aug 10 04:45:52 dist-test-slave-8hhm krb5kdc[17034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337151, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:45:52.402521 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.390108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.14.237.1:33955)
0810 04:45:52.390183 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:52.390185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:52.393416 (+ 3231us) server_negotiation.cc:407] Connection header received
0810 04:45:52.393449 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:52.393450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:52.393464 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:52.393532 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:52.394147 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.394897 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.395784 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.396088 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.396161 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.399102 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:52.399112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:52.399112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:52.399120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:52.401126 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:52.401579 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:52.401583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:52.401583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:52.401678 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:52.401960 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:52.401961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:52.401961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:52.402143 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:52.402287 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:52.402347 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 04:45:52.402424 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 04:45:52.402779 17434 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45178
I20260810 04:45:52.410207 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:52.403786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.14.237.62:37355)
0810 04:45:52.403830 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:52.403832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:52.403858 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:45:52.403930 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:52.403931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:52.403939 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:52.403984 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:52.404554 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.405033 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.406022 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:52.406240 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:52.406300 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:52.407563 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:52.407564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:52.407565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:52.407571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:52.409095 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:52.409524 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:52.409525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:52.409525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:52.409573 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:52.409773 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:52.409773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:52.409773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:52.409861 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:52.410004 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:52.410084 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 04:45:52.410147 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 04:45:52.410599 17115 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40356
I20260810 04:45:52.411422 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 17241
I20260810 04:45:52.430368 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 17479
I20260810 04:45:52.444738 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 17713
I20260810 04:45:52.457775 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 17059
2026-08-10T04:45:52Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2535 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:45:52 dist-test-slave-8hhm krb5kdc[18093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:45:52 dist-test-slave-8hhm krb5kdc[18093](info): set up 2 sockets
Aug 10 04:45:52 dist-test-slave-8hhm krb5kdc[18093](info): commencing operation
krb5kdc: starting...
W20260810 04:45:53.708101 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:45:53 dist-test-slave-8hhm krb5kdc[18093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:45:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:45:53.824179 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35585
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:35585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:53.905051 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:53.905313 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:53.905354 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:53.906426 18117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:45:53.906490 18117 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:45:53.906512 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:53.906533 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:53.906565 18117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:45:53.906589 18117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:45:53.908038 18117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:35585
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:53.908296 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:53.908504 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:53.913322 18131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:53.913331 18127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:53.913341 18125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:53.913707 18117 server_base.cc:1034] running on GCE node
I20260810 04:45:53.914120 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:53.914371 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:53.915551 18117 hybrid_clock.cc:648] HybridClock initialized: now 1786337153915536 us; error 47 us; skew 500 ppm
Aug 10 04:45:53 dist-test-slave-8hhm krb5kdc[18093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:53.921411 18117 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:45:53.921800 18117 webserver.cc:460] Webserver started at http://127.14.237.62:46295/ using document root <none> and password file <none>
I20260810 04:45:53.922014 18117 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:53.922056 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:53.922150 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:53.923135 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "fb75b362746a4314bf9919495be369bc"
format_stamp: "Formatted at 2026-08-10 04:45:53 on dist-test-slave-8hhm"
server_key: "71deaa4c40eb71333a5a770bae760795"
server_key_iv: "0ac75d7141052da42fbeb678ea26d903"
server_key_version: "encryptionkey@0"
I20260810 04:45:53.923432 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "fb75b362746a4314bf9919495be369bc"
format_stamp: "Formatted at 2026-08-10 04:45:53 on dist-test-slave-8hhm"
server_key: "71deaa4c40eb71333a5a770bae760795"
server_key_iv: "0ac75d7141052da42fbeb678ea26d903"
server_key_version: "encryptionkey@0"
I20260810 04:45:53.924702 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:45:53.925472 18148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:53.925635 18117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:53.925731 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "fb75b362746a4314bf9919495be369bc"
format_stamp: "Formatted at 2026-08-10 04:45:53 on dist-test-slave-8hhm"
server_key: "71deaa4c40eb71333a5a770bae760795"
server_key_iv: "0ac75d7141052da42fbeb678ea26d903"
server_key_version: "encryptionkey@0"
I20260810 04:45:53.925788 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:53.932919 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:53.934691 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:53.934803 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:53.939146 18117 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:35585
I20260810 04:45:53.939244 18252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:35585 every 8 connection(s)
I20260810 04:45:53.939558 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:45:53.940243 18254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:53.942536 18254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: Bootstrap starting.
I20260810 04:45:53.943126 18254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:53.943408 18254 log.cc:824] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:53.944129 18254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: No bootstrap required, opened a new log
I20260810 04:45:53.946386 18254 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75b362746a4314bf9919495be369bc" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35585 } }
I20260810 04:45:53.946538 18254 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:53.946576 18254 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb75b362746a4314bf9919495be369bc, State: Initialized, Role: FOLLOWER
I20260810 04:45:53.946736 18254 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75b362746a4314bf9919495be369bc" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35585 } }
I20260810 04:45:53.946818 18254 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:45:53.946854 18254 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:45:53.946902 18254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:53.947403 18254 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75b362746a4314bf9919495be369bc" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35585 } }
I20260810 04:45:53.947518 18254 leader_election.cc:302] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb75b362746a4314bf9919495be369bc; no voters:
I20260810 04:45:53.947705 18254 leader_election.cc:288] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:45:53.947739 18264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:53.947909 18264 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [term 1 LEADER]: Becoming Leader. State: Replica: fb75b362746a4314bf9919495be369bc, State: Running, Role: LEADER
I20260810 04:45:53.947948 18254 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:45:53.947997 18264 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75b362746a4314bf9919495be369bc" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35585 } }
I20260810 04:45:53.948395 18267 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb75b362746a4314bf9919495be369bc. Latest consensus state: current_term: 1 leader_uuid: "fb75b362746a4314bf9919495be369bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75b362746a4314bf9919495be369bc" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35585 } } }
I20260810 04:45:53.948390 18266 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb75b362746a4314bf9919495be369bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb75b362746a4314bf9919495be369bc" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35585 } } }
I20260810 04:45:53.948498 18267 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [sys.catalog]: This master's current role is: LEADER
I20260810 04:45:53.948546 18266 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc [sys.catalog]: This master's current role is: LEADER
I20260810 04:45:53.948792 18279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:45:53.948926 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18117
I20260810 04:45:53.949023 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:45:53.949239 15284 external_mini_cluster.cc:1428] Setting key 5bf480666ac15b1910705d21845c2dbf
I20260810 04:45:53.949404 18279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:45:53.951020 18279 catalog_manager.cc:1347] Generated new cluster ID: becc9489335541bd8b235353b75175dd
I20260810 04:45:53.951073 18279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:45:53 dist-test-slave-8hhm krb5kdc[18093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:53.961112 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:53.949924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.14.237.62:35585)
0810 04:45:53.950127 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:53.950135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:53.950328 (+ 193us) server_negotiation.cc:407] Connection header received
0810 04:45:53.950459 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:53.950466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:53.950523 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:53.950745 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:53.951515 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:53.952919 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:53.953943 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:53.954530 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:53.954681 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:53.957504 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:53.957508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:53.957508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:53.957520 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:53.959467 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:53.959886 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:53.959889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:53.959889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:53.960000 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:53.960227 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:53.960228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:53.960228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:53.960510 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:53.960648 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:53.960780 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 04:45:53.960905 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260810 04:45:53.979713 18279 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:45:53.980237 18279 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:45:53.985026 18279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: Generated new TSK 0
I20260810 04:45:53.985205 18279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:45:54.012234 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35585
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:54.094789 18301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:54.094991 18301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:54.095014 18301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:54.096076 18301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:54.096108 18301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:54.096128 18301 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:45:54.096139 18301 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:45:54.096158 18301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:45:54.097690 18301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:54.097880 18301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:54.098055 18301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:45:54.102978 18301 server_base.cc:1034] running on GCE node
W20260810 04:45:54.102962 18310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:54.103003 18313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:54.102946 18309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:54.103453 18301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:54.103701 18301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:54.104884 18301 hybrid_clock.cc:648] HybridClock initialized: now 1786337154104864 us; error 33 us; skew 500 ppm
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:54.107453 18301 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:45:54.107898 18301 webserver.cc:460] Webserver started at http://127.14.237.1:43561/ using document root <none> and password file <none>
I20260810 04:45:54.108163 18301 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:54.108240 18301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:54.108407 18301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:54.109529 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "d2dbcb067815446b9ee3ca818a024e66"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "8689ed97ce575a3f2d9f9bf74a30d4a0"
server_key_iv: "015d43aa49933dfe1b16ef685cbffce0"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.109907 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "d2dbcb067815446b9ee3ca818a024e66"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "8689ed97ce575a3f2d9f9bf74a30d4a0"
server_key_iv: "015d43aa49933dfe1b16ef685cbffce0"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.111348 18301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:54.112090 18326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:54.112300 18301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:54.112396 18301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "d2dbcb067815446b9ee3ca818a024e66"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "8689ed97ce575a3f2d9f9bf74a30d4a0"
server_key_iv: "015d43aa49933dfe1b16ef685cbffce0"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.112481 18301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:54.144495 18301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:54.146616 18301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:54.146803 18301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:54.147485 18301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:54.147537 18301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:54.147567 18301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:54.147606 18301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:54.155189 18301 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:41995
I20260810 04:45:54.155297 18534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:41995 every 8 connection(s)
I20260810 04:45:54.155596 18301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:45:54.157552 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18301
I20260810 04:45:54.157684 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:45:54.157838 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.147994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:36473 (local address 127.14.237.62:35585)
0810 04:45:54.148063 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.148066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.148299 (+ 233us) server_negotiation.cc:407] Connection header received
0810 04:45:54.148893 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.148896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.148914 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.148988 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:54.149838 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.150351 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.151395 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.151661 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.151738 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.154780 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:54.154781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:54.154782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:54.154788 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:54.156412 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.156843 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.156843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.156844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.156919 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.157127 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.157128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.157128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.157294 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:54.157387 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.157629 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.157736 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:45:54.157922 15284 external_mini_cluster.cc:1428] Setting key aca3c7bde47d701507b5b1dd601afe8a
I20260810 04:45:54.158010 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.147969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35585 (local address 127.14.237.1:36473)
0810 04:45:54.148155 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.148197 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.148794 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.149100 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.149105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.149450 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:54.149738 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.149745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.150457 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.150458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.151279 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.151281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.151759 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.151760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.151924 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.152471 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:54.152480 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:54.154683 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:54.156486 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.156491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.156492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.156781 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.156976 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.156977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.156977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.157077 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.157378 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:54.157382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:54.157492 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.157574 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.157786 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260810 04:45:54.158578 18538 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35585
I20260810 04:45:54.158692 18538 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:54.158890 18538 heartbeater.cc:511] Master 127.14.237.62:35585 requested a full tablet report, sending...
I20260810 04:45:54.159461 18176 ts_manager.cc:194] Registered new tserver with Master: d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995)
I20260810 04:45:54.160319 18176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:36473
I20260810 04:45:54.166028 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.159774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:56619 (local address 127.14.237.62:35585)
0810 04:45:54.159812 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.159814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.159915 (+ 101us) server_negotiation.cc:407] Connection header received
0810 04:45:54.159968 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.159969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.159978 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.160022 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:54.160485 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.161200 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.162230 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.162434 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.162530 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.163445 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:54.163446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:54.163447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:54.163454 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:54.164830 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.165243 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.165244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.165244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.165304 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.165513 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.165514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.165515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.165628 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:54.165698 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.165874 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.165946 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:45:54.166040 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.159722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35585 (local address 127.14.237.1:56619)
0810 04:45:54.159845 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.159854 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.159910 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.160054 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.160055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.160213 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:54.160304 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.160306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.161427 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.161428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.162135 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.162138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.162533 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.162534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.162605 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.163056 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:54.163060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:54.163355 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:54.164942 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.164944 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.164944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.165117 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.165380 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.165381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.165382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.165441 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.165703 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:54.165704 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:54.165746 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.165837 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.165941 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:45:54.186311 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35585
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:54.283597 18553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:54.283807 18553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:54.283831 18553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:54.284899 18553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:54.284933 18553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:54.284952 18553 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:45:54.284965 18553 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:45:54.284981 18553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:45:54.286430 18553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:54.286618 18553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:54.286795 18553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:54.291932 18560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:54.292053 18553 server_base.cc:1034] running on GCE node
W20260810 04:45:54.291963 18565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:54.292241 18561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:54.292462 18553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:54.292709 18553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:54.293885 18553 hybrid_clock.cc:648] HybridClock initialized: now 1786337154293871 us; error 29 us; skew 500 ppm
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:54.296401 18553 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:45:54.296806 18553 webserver.cc:460] Webserver started at http://127.14.237.2:37785/ using document root <none> and password file <none>
I20260810 04:45:54.297025 18553 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:54.297111 18553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:54.297361 18553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:54.298458 18553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "eaf6501c861b45749957c81dac1ad999"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "33184c2cf2ab94ff26c533d8441a6ac0"
server_key_iv: "ba95fa106fddfb45ecf3011ca5d21503"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.298816 18553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "eaf6501c861b45749957c81dac1ad999"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "33184c2cf2ab94ff26c533d8441a6ac0"
server_key_iv: "ba95fa106fddfb45ecf3011ca5d21503"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.300187 18553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:45:54.300937 18576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:54.301137 18553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:54.301263 18553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "eaf6501c861b45749957c81dac1ad999"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "33184c2cf2ab94ff26c533d8441a6ac0"
server_key_iv: "ba95fa106fddfb45ecf3011ca5d21503"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.301368 18553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:54.308872 18553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:54.311034 18553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:54.311184 18553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:54.311697 18553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:54.311762 18553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:54.311815 18553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:54.311844 18553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:54.318281 18553 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:40237
I20260810 04:45:54.318351 18760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:40237 every 8 connection(s)
I20260810 04:45:54.318653 18553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:54.321557 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.312184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:54293 (local address 127.14.237.62:35585)
0810 04:45:54.312256 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.312258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.312447 (+ 189us) server_negotiation.cc:407] Connection header received
0810 04:45:54.312967 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.312969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.312983 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.313035 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:54.313921 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.314747 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.316301 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.316615 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.316687 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.319199 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:54.319200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:54.319200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:54.319206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:54.320298 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.320666 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.320667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.320668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.320718 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.320940 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.320940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.320940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.321059 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:54.321115 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.321430 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.321495 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:45:54.321702 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.312172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35585 (local address 127.14.237.2:54293)
0810 04:45:54.312332 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.312365 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.312860 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.313114 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.313118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.313516 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:54.313803 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.313809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.314857 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.314858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.316205 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.316208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.316700 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.316701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.316797 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.317327 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:54.317338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:54.319106 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:54.320367 (+ 1261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.320373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.320374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.320583 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.320787 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.320787 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.320788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.320884 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.321118 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:54.321122 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:54.321298 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.321399 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.321518 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260810 04:45:54.322115 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18553
I20260810 04:45:54.322234 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:45:54.322285 18761 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35585
I20260810 04:45:54.322383 18761 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:54.322438 15284 external_mini_cluster.cc:1428] Setting key 19326606d881bed50cef19f26e3040ea
I20260810 04:45:54.322594 18761 heartbeater.cc:511] Master 127.14.237.62:35585 requested a full tablet report, sending...
I20260810 04:45:54.322934 18176 ts_manager.cc:194] Registered new tserver with Master: eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237)
I20260810 04:45:54.323381 18176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:54293
I20260810 04:45:54.329861 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.323816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:33261 (local address 127.14.237.62:35585)
0810 04:45:54.323855 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.323857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.323937 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:45:54.324001 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.324002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.324009 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.324049 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:54.324457 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.324900 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.325862 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.326068 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.326143 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.327183 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:54.327184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:54.327184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:54.327190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:54.328595 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.328972 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.328974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.328974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.329055 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.329297 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.329298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.329299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.329452 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:54.329529 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.329693 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.329770 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:45:54.329873 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.323723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35585 (local address 127.14.237.2:33261)
0810 04:45:54.323871 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.323883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.323945 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.324086 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.324087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.324324 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:54.324388 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.324390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.324997 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.324998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.325758 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.325761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.326143 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.326144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.326247 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.326686 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:54.326691 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:54.327095 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:54.328693 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.328694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.328695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.328893 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.329136 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.329137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.329137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.329198 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.329534 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:54.329535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:54.329581 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.329666 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.329771 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:45:54.349920 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35585
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:45:54.433492 18774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:45:54.433715 18774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:45:54.433756 18774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:45:54.434865 18774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:45:54.434937 18774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:45:54.434969 18774 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:45:54.434995 18774 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:45:54.435026 18774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:45:54.436470 18774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:35585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:45:54.436749 18774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:45:54.436976 18774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:45:54.442169 18784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:54.442158 18781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:45:54.442160 18780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:45:54.442412 18774 server_base.cc:1034] running on GCE node
I20260810 04:45:54.442651 18774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:45:54.442893 18774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:45:54.444069 18774 hybrid_clock.cc:648] HybridClock initialized: now 1786337154444058 us; error 29 us; skew 500 ppm
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:45:54.446657 18774 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:45:54.447103 18774 webserver.cc:460] Webserver started at http://127.14.237.3:42761/ using document root <none> and password file <none>
I20260810 04:45:54.447340 18774 fs_manager.cc:359] Metadata directory not provided
I20260810 04:45:54.447408 18774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:45:54.447548 18774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:45:54.448745 18774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "7c75387a86284e6c9acf15e94887c6d7"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "12bf4e1f8aba2f78e876b8fce10b7635"
server_key_iv: "2b345c4ec201b276960d434012046b0e"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.449134 18774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "7c75387a86284e6c9acf15e94887c6d7"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "12bf4e1f8aba2f78e876b8fce10b7635"
server_key_iv: "2b345c4ec201b276960d434012046b0e"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.450711 18774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:54.451448 18795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:54.451632 18774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:54.451732 18774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "7c75387a86284e6c9acf15e94887c6d7"
format_stamp: "Formatted at 2026-08-10 04:45:54 on dist-test-slave-8hhm"
server_key: "12bf4e1f8aba2f78e876b8fce10b7635"
server_key_iv: "2b345c4ec201b276960d434012046b0e"
server_key_version: "encryptionkey@0"
I20260810 04:45:54.451835 18774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:45:54.463332 18774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:45:54.465116 18774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:45:54.465332 18774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:45:54.465840 18774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:45:54.465916 18774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:54.465960 18774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:45:54.465989 18774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:54.473166 18774 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:46349
I20260810 04:45:54.473281 18986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:46349 every 8 connection(s)
I20260810 04:45:54.473572 18774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:45:54.474897 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18774
I20260810 04:45:54.474998 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:45:54.475199 15284 external_mini_cluster.cc:1428] Setting key 38956435a0900552c25c92d6cb215c1f
I20260810 04:45:54.476125 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.466385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:34799 (local address 127.14.237.62:35585)
0810 04:45:54.466427 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.466429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.466651 (+ 222us) server_negotiation.cc:407] Connection header received
0810 04:45:54.467168 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.467169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.467182 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.467229 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:54.468000 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.468822 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.469923 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.470102 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.470177 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.472932 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:54.472934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:54.472934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:54.472942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:54.474507 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.474952 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.474954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.474954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.475043 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.475348 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.475349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.475349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.475504 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:54.475595 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.475929 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.476025 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:45:54.476317 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.466382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35585 (local address 127.14.237.3:34799)
0810 04:45:54.466532 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.466566 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.467051 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.467298 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.467301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.467609 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:54.467883 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.467889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.468938 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.468939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.469828 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.469830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.470195 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.470196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.470318 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.470877 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:54.470888 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:54.472838 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:54.474620 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.474627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.474629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.474859 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.475104 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.475105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.475105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.475267 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.475584 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:54.475590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:54.475776 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.475893 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.476070 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260810 04:45:54.476820 18987 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35585
I20260810 04:45:54.476898 18987 heartbeater.cc:464] Registering TS with master...
I20260810 04:45:54.477094 18987 heartbeater.cc:511] Master 127.14.237.62:35585 requested a full tablet report, sending...
I20260810 04:45:54.477516 18174 ts_manager.cc:194] Registered new tserver with Master: 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:54.477977 18174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:34799
I20260810 04:45:54.483402 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.477974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:51569 (local address 127.14.237.62:35585)
0810 04:45:54.478011 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.478013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.478096 (+ 83us) server_negotiation.cc:407] Connection header received
0810 04:45:54.478171 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.478172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.478182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.478245 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:54.478570 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.479021 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.479696 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.479923 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.480001 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.480809 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:54.480810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:54.480811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:54.480819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:54.482107 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.482519 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.482520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.482521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.482594 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.482813 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.482814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.482815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.482930 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:54.483020 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.483226 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.483307 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:45:54.483417 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.477879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35585 (local address 127.14.237.3:51569)
0810 04:45:54.478025 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.478038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.478121 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.478263 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.478263 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.478435 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:54.478494 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.478495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.479114 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.479115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.479611 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.479613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.480000 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.480001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.480081 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.480443 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:54.480447 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:54.480700 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:54.482221 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.482222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.482223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.482422 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.482669 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.482670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.482670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.482740 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.483011 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:54.483012 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:54.483106 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.483197 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.483307 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260810 04:45:54.486814 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:54.510838 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.503284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.14.237.62:35585)
0810 04:45:54.503353 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.503355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.503383 (+ 28us) server_negotiation.cc:407] Connection header received
0810 04:45:54.503412 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.503413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.503422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.503465 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:54.504109 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.504599 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.505434 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.505612 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.505667 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.508205 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:54.508206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:54.508207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:54.508231 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:54.509700 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.510052 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.510053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.510054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.510116 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.510321 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.510322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.510322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.510439 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:54.510543 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.510652 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.510735 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:45:54.512409 18174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:45:54.514011 18174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:54.520916 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.516796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.14.237.3:46349)
0810 04:45:54.517004 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.517008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.517017 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:45:54.517051 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.517053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.517109 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.517192 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:54.517379 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.518436 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.519732 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.520476 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.520562 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.520623 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.520766 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.520815 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
I20260810 04:45:54.521131 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.516566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.14.237.1:41995)
0810 04:45:54.517651 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.517655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.517663 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:45:54.517698 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.517700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.517713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.517790 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:54.517897 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.518861 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.520062 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.520572 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.520643 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.520702 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.520981 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.521034 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":111,"threads_started":1}
I20260810 04:45:54.521142 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.516590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.14.237.2:40237)
0810 04:45:54.517389 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.517393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.517401 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:45:54.517441 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.517443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.517457 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.517531 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:54.517642 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.518413 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.519842 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.520511 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.520570 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.520623 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.520994 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.521045 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":92,"threads_started":1}
I20260810 04:45:54.520946 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.516676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:46349 (local address 127.0.0.1:43054)
0810 04:45:54.516797 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.516841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.516910 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.517206 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.517234 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.517237 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:54.517303 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.517306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.518531 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.518532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.519628 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.519630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.520571 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.520572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.520653 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.520667 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.520742 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.520799 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":28,"thread_start_us":75,"threads_started":1}
I20260810 04:45:54.521440 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.516477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40237 (local address 127.0.0.1:54338)
0810 04:45:54.516801 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.516843 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.516910 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.517528 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.517528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.517530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:54.517567 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.517568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.518502 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.518503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.519757 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.519760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.520660 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.520661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.520763 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.520767 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.520819 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.520859 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":43,"thread_start_us":80,"threads_started":1}
I20260810 04:45:54.521517 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.516465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41995 (local address 127.0.0.1:42338)
0810 04:45:54.516797 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.516841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.516910 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.517803 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.517804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.517806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:54.517839 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.517840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.518945 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.518946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.519978 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.519980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.520746 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.520747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.520802 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.520805 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.520853 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.520893 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":57,"threads_started":1}
I20260810 04:45:54.522269 18411 tablet_service.cc:1467] Processing CreateTablet for tablet 11d1d0e93d0043eebfab542ae75067d0 (DEFAULT_TABLE table=test-table [id=53662447d0124f8397d5600b2b1d74e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:54.522267 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 11d1d0e93d0043eebfab542ae75067d0 (DEFAULT_TABLE table=test-table [id=53662447d0124f8397d5600b2b1d74e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:54.522266 18661 tablet_service.cc:1467] Processing CreateTablet for tablet 11d1d0e93d0043eebfab542ae75067d0 (DEFAULT_TABLE table=test-table [id=53662447d0124f8397d5600b2b1d74e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:54.522660 18886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d1d0e93d0043eebfab542ae75067d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:54.522660 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d1d0e93d0043eebfab542ae75067d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:54.522678 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d1d0e93d0043eebfab542ae75067d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:54.525645 19026 tablet_bootstrap.cc:492] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Bootstrap starting.
I20260810 04:45:54.526106 19029 tablet_bootstrap.cc:492] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Bootstrap starting.
I20260810 04:45:54.526360 19033 tablet_bootstrap.cc:492] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Bootstrap starting.
I20260810 04:45:54.526695 19026 tablet_bootstrap.cc:654] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:54.526959 19029 tablet_bootstrap.cc:654] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:54.526984 19033 tablet_bootstrap.cc:654] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:54.527179 19026 log.cc:824] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:54.527197 19029 log.cc:824] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:54.527315 19033 log.cc:824] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Log is configured to *not* fsync() on all Append() calls
I20260810 04:45:54.528033 19033 tablet_bootstrap.cc:492] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: No bootstrap required, opened a new log
I20260810 04:45:54.528065 19029 tablet_bootstrap.cc:492] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: No bootstrap required, opened a new log
I20260810 04:45:54.528101 19033 ts_tablet_manager.cc:1397] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:54.528132 19029 ts_tablet_manager.cc:1397] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:45:54.528218 19026 tablet_bootstrap.cc:492] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: No bootstrap required, opened a new log
I20260810 04:45:54.528276 19026 ts_tablet_manager.cc:1397] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:45:54.529716 19029 raft_consensus.cc:348] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.529870 19029 raft_consensus.cc:374] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:54.529915 19029 raft_consensus.cc:729] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c75387a86284e6c9acf15e94887c6d7, State: Initialized, Role: FOLLOWER
I20260810 04:45:54.530030 19029 consensus_queue.cc:260] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.530120 19033 raft_consensus.cc:348] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.530289 19033 raft_consensus.cc:374] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:54.530229 19026 raft_consensus.cc:348] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.530334 19033 raft_consensus.cc:729] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf6501c861b45749957c81dac1ad999, State: Initialized, Role: FOLLOWER
I20260810 04:45:54.530333 19029 ts_tablet_manager.cc:1428] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:45:54.530452 18987 heartbeater.cc:503] Master 127.14.237.62:35585 was elected leader, sending a full tablet report...
I20260810 04:45:54.530470 19033 consensus_queue.cc:260] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.530376 19026 raft_consensus.cc:374] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:54.530750 19033 ts_tablet_manager.cc:1428] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:45:54.530737 19026 raft_consensus.cc:729] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2dbcb067815446b9ee3ca818a024e66, State: Initialized, Role: FOLLOWER
I20260810 04:45:54.530896 18761 heartbeater.cc:503] Master 127.14.237.62:35585 was elected leader, sending a full tablet report...
I20260810 04:45:54.530882 19026 consensus_queue.cc:260] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.531216 18538 heartbeater.cc:503] Master 127.14.237.62:35585 was elected leader, sending a full tablet report...
I20260810 04:45:54.531260 19026 ts_tablet_manager.cc:1428] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260810 04:45:54.569368 18762 tablet.cc:2369] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:45:54.656381 18540 tablet.cc:2369] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:45:54.724846 18988 tablet.cc:2369] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:45:54.873770 19041 raft_consensus.cc:482] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:54.873925 19041 raft_consensus.cc:504] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.874329 19041 leader_election.cc:288] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237), 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:54.877480 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.874619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:53741 (local address 127.14.237.2:40237)
0810 04:45:54.874691 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.874693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.874781 (+ 88us) server_negotiation.cc:407] Connection header received
0810 04:45:54.874894 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.874895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.874908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.874964 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:54.875173 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.875659 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.876744 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.877096 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.877177 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.877185 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.877366 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.877404 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:45:54.877497 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.874514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40237 (local address 127.14.237.1:53741)
0810 04:45:54.874692 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.874705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.874820 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.874982 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.874982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.874985 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:54.875087 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.875090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.875742 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.875743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.876636 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.876639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.877177 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.877178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.877273 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.877276 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.877357 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.877404 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260810 04:45:54.877497 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.874796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:34147 (local address 127.14.237.3:46349)
0810 04:45:54.874872 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.874874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.874884 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:45:54.874974 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.874975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.874984 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.875042 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:54.875198 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.875675 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.876755 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.877112 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.877197 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.877206 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.877384 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.877422 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 04:45:54.877506 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.874698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:46349 (local address 127.14.237.1:34147)
0810 04:45:54.874807 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.874837 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.874897 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.875045 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.875046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.875047 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:54.875122 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.875124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.875761 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.875762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.876641 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.876644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.877199 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.877200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.877295 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.877298 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.877367 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.877409 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":31,"thread_start_us":42,"threads_started":1}
I20260810 04:45:54.877871 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d1d0e93d0043eebfab542ae75067d0" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75387a86284e6c9acf15e94887c6d7" is_pre_election: true
I20260810 04:45:54.877907 18691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d1d0e93d0043eebfab542ae75067d0" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf6501c861b45749957c81dac1ad999" is_pre_election: true
I20260810 04:45:54.878029 18916 raft_consensus.cc:2454] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 0.
I20260810 04:45:54.878047 18691 raft_consensus.cc:2454] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 0.
I20260810 04:45:54.878291 18329 leader_election.cc:302] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c75387a86284e6c9acf15e94887c6d7, d2dbcb067815446b9ee3ca818a024e66; no voters:
I20260810 04:45:54.878448 19041 raft_consensus.cc:2781] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:54.878505 19041 raft_consensus.cc:482] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:54.878535 19041 raft_consensus.cc:3037] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:54.879205 19041 raft_consensus.cc:504] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.879336 19041 leader_election.cc:288] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 election: Requested vote from peers eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237), 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:54.879489 18691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d1d0e93d0043eebfab542ae75067d0" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf6501c861b45749957c81dac1ad999"
I20260810 04:45:54.879491 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d1d0e93d0043eebfab542ae75067d0" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75387a86284e6c9acf15e94887c6d7"
I20260810 04:45:54.879570 18691 raft_consensus.cc:3037] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:54.879590 18916 raft_consensus.cc:3037] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:54.880311 18691 raft_consensus.cc:2454] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 1.
I20260810 04:45:54.880318 18916 raft_consensus.cc:2454] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 1.
I20260810 04:45:54.880466 18329 leader_election.cc:302] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2dbcb067815446b9ee3ca818a024e66, eaf6501c861b45749957c81dac1ad999; no voters:
I20260810 04:45:54.880568 19041 raft_consensus.cc:2781] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:54.880693 19041 raft_consensus.cc:686] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 1 LEADER]: Becoming Leader. State: Replica: d2dbcb067815446b9ee3ca818a024e66, State: Running, Role: LEADER
I20260810 04:45:54.880785 19041 consensus_queue.cc:237] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.881722 18175 catalog_manager.cc:5495] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1). New cstate: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } health_report { overall_health: UNKNOWN } } }
Aug 10 04:45:54 dist-test-slave-8hhm krb5kdc[18093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:54.988791 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.981944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35585 (local address 127.0.0.1:50244)
0810 04:45:54.982111 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:54.982122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:54.982192 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:54.982333 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:54.982334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:54.982644 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:45:54.982695 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.982697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.983357 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.983358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.983876 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:54.983878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.984217 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.984217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:54.984260 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.984742 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:45:54.984746 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:45:54.986287 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:45:54.987900 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.987906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.987908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.988089 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.988267 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:45:54.988268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:45:54.988268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:45:54.988323 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:45:54.988521 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:45:54.988527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:45:54.988565 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 04:45:54.988609 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 04:45:54.988695 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260810 04:45:54.988832 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:54.982037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.14.237.62:35585)
0810 04:45:54.982125 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:54.982127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:54.982186 (+ 59us) server_negotiation.cc:407] Connection header received
0810 04:45:54.982250 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:54.982251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:54.982260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:54.982315 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:54.982772 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.983288 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.983942 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:54.984149 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:54.984204 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:54.986368 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:54.986369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:54.986369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:54.986375 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:54.987834 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.988156 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.988157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.988157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.988209 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:54.988372 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:54.988373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:54.988373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:54.988469 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:54.988517 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:54.988631 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 04:45:54.988739 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":48}
I20260810 04:45:54.990923 18175 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:50244:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:45:54.991186 18175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:54.992753 18886 tablet_service.cc:1467] Processing CreateTablet for tablet f0f8cd7566514f85a2beaafb49c2f341 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b480326388cc497c8cfcfe4c33b132b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:45:54.992794 18411 tablet_service.cc:1467] Processing CreateTablet for tablet f0f8cd7566514f85a2beaafb49c2f341 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b480326388cc497c8cfcfe4c33b132b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:45:54.992946 18886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f8cd7566514f85a2beaafb49c2f341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:54.992995 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f8cd7566514f85a2beaafb49c2f341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:54.994131 19029 tablet_bootstrap.cc:492] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: Bootstrap starting.
I20260810 04:45:54.994355 19026 tablet_bootstrap.cc:492] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: Bootstrap starting.
I20260810 04:45:54.994469 18661 tablet_service.cc:1467] Processing CreateTablet for tablet f0f8cd7566514f85a2beaafb49c2f341 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b480326388cc497c8cfcfe4c33b132b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:45:54.994534 19029 tablet_bootstrap.cc:654] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:54.994632 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f8cd7566514f85a2beaafb49c2f341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:54.994733 19026 tablet_bootstrap.cc:654] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:54.995265 19029 tablet_bootstrap.cc:492] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: No bootstrap required, opened a new log
I20260810 04:45:54.995322 19029 ts_tablet_manager.cc:1397] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:54.995484 19029 raft_consensus.cc:348] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.995579 19029 raft_consensus.cc:374] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:54.995599 19029 raft_consensus.cc:729] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c75387a86284e6c9acf15e94887c6d7, State: Initialized, Role: FOLLOWER
I20260810 04:45:54.995640 19029 consensus_queue.cc:260] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.995743 19029 ts_tablet_manager.cc:1428] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:54.995882 19039 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:54.995970 19039 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:54.996212 19033 tablet_bootstrap.cc:492] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: Bootstrap starting.
I20260810 04:45:54.996611 19026 tablet_bootstrap.cc:492] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: No bootstrap required, opened a new log
I20260810 04:45:54.996620 19033 tablet_bootstrap.cc:654] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:54.996656 19026 ts_tablet_manager.cc:1397] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:54.996807 19026 raft_consensus.cc:348] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.996877 19026 raft_consensus.cc:374] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:54.996903 19026 raft_consensus.cc:729] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2dbcb067815446b9ee3ca818a024e66, State: Initialized, Role: FOLLOWER
I20260810 04:45:54.996950 19026 consensus_queue.cc:260] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.997037 19026 ts_tablet_manager.cc:1428] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:54.997170 19041 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:54.997270 19041 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:54.997437 19033 tablet_bootstrap.cc:492] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: No bootstrap required, opened a new log
I20260810 04:45:54.997494 19033 ts_tablet_manager.cc:1397] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:54.997674 19033 raft_consensus.cc:348] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.997764 19033 raft_consensus.cc:374] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:54.997786 19033 raft_consensus.cc:729] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf6501c861b45749957c81dac1ad999, State: Initialized, Role: FOLLOWER
I20260810 04:45:54.997838 19033 consensus_queue.cc:260] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:54.997936 19033 ts_tablet_manager.cc:1428] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:54.998008 19040 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:54.998075 19040 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:55.139846 19041 raft_consensus.cc:482] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:55.139950 19041 raft_consensus.cc:504] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.140146 19041 leader_election.cc:288] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237), 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:55.140576 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f8cd7566514f85a2beaafb49c2f341" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75387a86284e6c9acf15e94887c6d7" is_pre_election: true
I20260810 04:45:55.140559 18691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f8cd7566514f85a2beaafb49c2f341" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf6501c861b45749957c81dac1ad999" is_pre_election: true
I20260810 04:45:55.140691 18691 raft_consensus.cc:2454] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 0.
I20260810 04:45:55.140691 18916 raft_consensus.cc:2454] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 0.
I20260810 04:45:55.140856 18329 leader_election.cc:302] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2dbcb067815446b9ee3ca818a024e66, eaf6501c861b45749957c81dac1ad999; no voters:
I20260810 04:45:55.140985 19041 raft_consensus.cc:2781] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:55.141058 19041 raft_consensus.cc:482] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:55.141083 19041 raft_consensus.cc:3037] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:55.141778 19041 raft_consensus.cc:504] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.141913 19041 leader_election.cc:288] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 election: Requested vote from peers eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237), 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:55.142153 18691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f8cd7566514f85a2beaafb49c2f341" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf6501c861b45749957c81dac1ad999"
I20260810 04:45:55.142234 18691 raft_consensus.cc:3037] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:55.142212 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0f8cd7566514f85a2beaafb49c2f341" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75387a86284e6c9acf15e94887c6d7"
I20260810 04:45:55.142323 18916 raft_consensus.cc:3037] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:55.142877 18691 raft_consensus.cc:2454] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 1.
I20260810 04:45:55.143040 18916 raft_consensus.cc:2454] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 1.
I20260810 04:45:55.143088 18329 leader_election.cc:302] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2dbcb067815446b9ee3ca818a024e66, eaf6501c861b45749957c81dac1ad999; no voters:
I20260810 04:45:55.143213 19041 raft_consensus.cc:2781] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:55.143280 19041 raft_consensus.cc:686] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [term 1 LEADER]: Becoming Leader. State: Replica: d2dbcb067815446b9ee3ca818a024e66, State: Running, Role: LEADER
I20260810 04:45:55.143345 19041 consensus_queue.cc:237] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.143695 19046 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: TxnStatusTablet state changed. Reason: New leader d2dbcb067815446b9ee3ca818a024e66. Latest consensus state: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:55.143798 19046 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:45:55.144042 18175 catalog_manager.cc:5495] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1). New cstate: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } health_report { overall_health: UNKNOWN } } }
I20260810 04:45:55.144384 19068 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:45:55.144531 19068 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:45:55.180742 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:55.177855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41995 (local address 127.0.0.1:42352)
0810 04:45:55.177935 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:55.177948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:55.178015 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:55.178289 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:55.178290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:55.178292 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:45:55.178349 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:55.178351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.179136 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.179137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:55.179723 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:55.179726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.180165 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.180166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:55.180234 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:55.180254 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:45:55.180550 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:45:55.180562 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 04:45:55.180609 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 04:45:55.180660 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:45:55.180754 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:55.177949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42352 (local address 127.14.237.1:41995)
0810 04:45:55.178120 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:55.178122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:55.178129 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:45:55.178175 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:55.178175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:55.178193 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:55.178281 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:55.178427 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.179047 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.179815 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.180073 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.180164 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:55.180366 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:55.180478 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:55.180572 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:55.180613 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 04:45:55.180635 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260810 04:45:55.182854 18916 raft_consensus.cc:1261] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Refusing update from remote peer d2dbcb067815446b9ee3ca818a024e66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:55.182854 18691 raft_consensus.cc:1261] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Refusing update from remote peer d2dbcb067815446b9ee3ca818a024e66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:55.183210 19041 consensus_queue.cc:1035] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:55.183367 19046 consensus_queue.cc:1035] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:55.184568 19039 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: TxnStatusTablet state changed. Reason: New leader d2dbcb067815446b9ee3ca818a024e66. Latest consensus state: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:55.184672 19039 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:55.184955 19040 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: TxnStatusTablet state changed. Reason: New leader d2dbcb067815446b9ee3ca818a024e66. Latest consensus state: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:55.185046 19040 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:55.185813 19046 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:55.185899 19046 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:45:55.185900 19039 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:55.185963 19075 mvcc.cc:205] Tried to move back new op lower bound from 7316836987625918464 to 7316836987467792384. Current Snapshot: MvccSnapshot[applied={T|T < 7316836987625918464}]
I20260810 04:45:55.185983 19039 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P 7c75387a86284e6c9acf15e94887c6d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:55.187042 19046 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:55.187134 19046 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P d2dbcb067815446b9ee3ca818a024e66: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:45:55.187443 19040 tablet_replica.cc:439] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } } }
I20260810 04:45:55.187522 19040 tablet_replica.cc:442] T f0f8cd7566514f85a2beaafb49c2f341 P eaf6501c861b45749957c81dac1ad999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:45:55.188369 19076 mvcc.cc:205] Tried to move back new op lower bound from 7316836987625918464 to 7316836987467792384. Current Snapshot: MvccSnapshot[applied={T|T < 7316836987625918464}]
I20260810 04:45:55.190739 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:55.188197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.14.237.1:41995)
0810 04:45:55.188236 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:55.188238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:55.188292 (+ 54us) server_negotiation.cc:407] Connection header received
0810 04:45:55.188359 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:55.188359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:55.188369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:55.188416 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:55.188667 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.189183 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.190045 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.190242 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.190316 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:55.190464 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:55.190497 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:55.190561 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:55.190621 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 04:45:55.190641 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:45:55.195618 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:55.193160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:52349 (local address 127.14.237.1:41995)
0810 04:45:55.193196 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:55.193197 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:55.193319 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:45:55.193354 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:55.193355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:55.193363 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:55.193397 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:55.193545 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.193939 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.195009 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.195166 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.195208 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:55.195422 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:55.195443 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:55.195478 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:55.195544 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 04:45:55.195560 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:45:55.195698 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:55.193055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41995 (local address 127.14.237.1:52349)
0810 04:45:55.193246 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:55.193257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:55.193312 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:55.193423 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:55.193424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:55.193425 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:45:55.193489 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:55.193490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.194009 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.194010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:55.194926 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:55.194929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:55.195237 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:55.195238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:55.195319 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:55.195334 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:45:55.195495 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:45:55.195496 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:45:55.195543 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 04:45:55.195580 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260810 04:45:55.197796 18691 raft_consensus.cc:1261] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Refusing update from remote peer d2dbcb067815446b9ee3ca818a024e66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:55.197796 18916 raft_consensus.cc:1261] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Refusing update from remote peer d2dbcb067815446b9ee3ca818a024e66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:55.197995 19096 consensus_queue.cc:1035] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:55.198102 19041 consensus_queue.cc:1035] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:55.199710 19080 mvcc.cc:205] Tried to move back new op lower bound from 7316836987689013248 to 7316836986392317952. Current Snapshot: MvccSnapshot[applied={T|T < 7316836987689013248}]
I20260810 04:45:55.212324 18176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:55.212450 18176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:55.213268 18176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: Sending DeleteTablet for 3 replicas of tablet 11d1d0e93d0043eebfab542ae75067d0
I20260810 04:45:55.213558 18411 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d1d0e93d0043eebfab542ae75067d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:55 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:42338
I20260810 04:45:55.213598 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d1d0e93d0043eebfab542ae75067d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:55 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:43054
I20260810 04:45:55.213666 18661 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d1d0e93d0043eebfab542ae75067d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:55 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:54338
I20260810 04:45:55.213835 18176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 04:45:55.213905 19105 tablet_replica.cc:330] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: stopping tablet replica
W20260810 04:45:55.213971 18176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:55.214162 19106 tablet_replica.cc:330] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: stopping tablet replica
I20260810 04:45:55.214308 19106 raft_consensus.cc:2229] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:55.214390 19107 tablet_replica.cc:330] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: stopping tablet replica
I20260810 04:45:55.214432 19105 raft_consensus.cc:2229] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:45:55.214501 19107 raft_consensus.cc:2229] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:55.214560 19107 raft_consensus.cc:2258] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:55.214681 19106 raft_consensus.cc:2258] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:55.214813 19105 raft_consensus.cc:2258] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:55.214982 19107 ts_tablet_manager.cc:1905] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:55.215317 19106 ts_tablet_manager.cc:1905] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:55.215441 18411 tablet_service.cc:1467] Processing CreateTablet for tablet 48f9c4343e934391aa28df42de2402b8 (DEFAULT_TABLE table=test-table [id=9f6b4aafae1749da96da6fb14939a549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:55.215585 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f9c4343e934391aa28df42de2402b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:55.216845 19107 ts_tablet_manager.cc:1918] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:55.216936 19107 log.cc:1196] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/11d1d0e93d0043eebfab542ae75067d0
I20260810 04:45:55.217070 19026 tablet_bootstrap.cc:492] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: Bootstrap starting.
I20260810 04:45:55.217253 19107 ts_tablet_manager.cc:1939] T 11d1d0e93d0043eebfab542ae75067d0 P eaf6501c861b45749957c81dac1ad999: Deleting consensus metadata
I20260810 04:45:55.217495 19026 tablet_bootstrap.cc:654] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:55.217635 19106 ts_tablet_manager.cc:1918] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:55.217738 19106 log.cc:1196] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/11d1d0e93d0043eebfab542ae75067d0
I20260810 04:45:55.218003 19106 ts_tablet_manager.cc:1939] T 11d1d0e93d0043eebfab542ae75067d0 P 7c75387a86284e6c9acf15e94887c6d7: Deleting consensus metadata
I20260810 04:45:55.218001 18151 catalog_manager.cc:4841] TS eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237): tablet 11d1d0e93d0043eebfab542ae75067d0 (table test-table [id=53662447d0124f8397d5600b2b1d74e5]) successfully deleted
I20260810 04:45:55.218048 18661 tablet_service.cc:1467] Processing CreateTablet for tablet 48f9c4343e934391aa28df42de2402b8 (DEFAULT_TABLE table=test-table [id=9f6b4aafae1749da96da6fb14939a549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:55.218142 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f9c4343e934391aa28df42de2402b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:55.218214 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 48f9c4343e934391aa28df42de2402b8 (DEFAULT_TABLE table=test-table [id=9f6b4aafae1749da96da6fb14939a549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:55.218307 18886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f9c4343e934391aa28df42de2402b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:55.218393 18151 catalog_manager.cc:4841] TS 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349): tablet 11d1d0e93d0043eebfab542ae75067d0 (table test-table [id=53662447d0124f8397d5600b2b1d74e5]) successfully deleted
I20260810 04:45:55.218657 19105 ts_tablet_manager.cc:1905] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:55.218851 19026 tablet_bootstrap.cc:492] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: No bootstrap required, opened a new log
I20260810 04:45:55.218899 19026 ts_tablet_manager.cc:1397] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:45:55.219007 19026 raft_consensus.cc:348] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.219070 19026 raft_consensus.cc:374] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:55.219089 19026 raft_consensus.cc:729] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2dbcb067815446b9ee3ca818a024e66, State: Initialized, Role: FOLLOWER
I20260810 04:45:55.219132 19026 consensus_queue.cc:260] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.219229 19026 ts_tablet_manager.cc:1428] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:55.219355 19033 tablet_bootstrap.cc:492] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: Bootstrap starting.
I20260810 04:45:55.219460 19029 tablet_bootstrap.cc:492] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: Bootstrap starting.
I20260810 04:45:55.219693 19033 tablet_bootstrap.cc:654] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:55.219852 19029 tablet_bootstrap.cc:654] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:55.220310 19033 tablet_bootstrap.cc:492] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: No bootstrap required, opened a new log
I20260810 04:45:55.220362 19033 ts_tablet_manager.cc:1397] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:55.220495 19033 raft_consensus.cc:348] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.220570 19033 raft_consensus.cc:374] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:55.220597 19033 raft_consensus.cc:729] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf6501c861b45749957c81dac1ad999, State: Initialized, Role: FOLLOWER
I20260810 04:45:55.220635 19033 consensus_queue.cc:260] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.220719 19033 ts_tablet_manager.cc:1428] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:45:55.220890 19105 ts_tablet_manager.cc:1918] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:55.220958 19105 log.cc:1196] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/11d1d0e93d0043eebfab542ae75067d0
I20260810 04:45:55.221297 19105 ts_tablet_manager.cc:1939] T 11d1d0e93d0043eebfab542ae75067d0 P d2dbcb067815446b9ee3ca818a024e66: Deleting consensus metadata
I20260810 04:45:55.221668 19029 tablet_bootstrap.cc:492] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: No bootstrap required, opened a new log
I20260810 04:45:55.221728 19029 ts_tablet_manager.cc:1397] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:55.221719 18152 catalog_manager.cc:4841] TS d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995): tablet 11d1d0e93d0043eebfab542ae75067d0 (table test-table [id=53662447d0124f8397d5600b2b1d74e5]) successfully deleted
I20260810 04:45:55.221879 19029 raft_consensus.cc:348] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.221971 19029 raft_consensus.cc:374] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:55.221999 19029 raft_consensus.cc:729] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c75387a86284e6c9acf15e94887c6d7, State: Initialized, Role: FOLLOWER
I20260810 04:45:55.222043 19029 consensus_queue.cc:260] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.222129 19029 ts_tablet_manager.cc:1428] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:55.289357 19041 raft_consensus.cc:482] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:55.289464 19041 raft_consensus.cc:504] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.289618 19041 leader_election.cc:288] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237), 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:55.289903 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9c4343e934391aa28df42de2402b8" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75387a86284e6c9acf15e94887c6d7" is_pre_election: true
I20260810 04:45:55.289956 18691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9c4343e934391aa28df42de2402b8" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf6501c861b45749957c81dac1ad999" is_pre_election: true
I20260810 04:45:55.290023 18916 raft_consensus.cc:2454] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 0.
I20260810 04:45:55.290057 18691 raft_consensus.cc:2454] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 0.
I20260810 04:45:55.290237 18329 leader_election.cc:302] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2dbcb067815446b9ee3ca818a024e66, eaf6501c861b45749957c81dac1ad999; no voters:
I20260810 04:45:55.290350 19041 raft_consensus.cc:2781] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:55.290402 19041 raft_consensus.cc:482] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:55.290428 19041 raft_consensus.cc:3037] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:55.291090 19041 raft_consensus.cc:504] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.291253 19041 leader_election.cc:288] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 election: Requested vote from peers eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237), 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:55.291472 18691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9c4343e934391aa28df42de2402b8" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf6501c861b45749957c81dac1ad999"
I20260810 04:45:55.291481 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9c4343e934391aa28df42de2402b8" candidate_uuid: "d2dbcb067815446b9ee3ca818a024e66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75387a86284e6c9acf15e94887c6d7"
I20260810 04:45:55.291554 18916 raft_consensus.cc:3037] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:55.291564 18691 raft_consensus.cc:3037] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:55.292237 18691 raft_consensus.cc:2454] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 1.
I20260810 04:45:55.292356 18916 raft_consensus.cc:2454] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dbcb067815446b9ee3ca818a024e66 in term 1.
I20260810 04:45:55.292441 18329 leader_election.cc:302] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2dbcb067815446b9ee3ca818a024e66, eaf6501c861b45749957c81dac1ad999; no voters:
I20260810 04:45:55.292521 19041 raft_consensus.cc:2781] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:55.292572 19041 raft_consensus.cc:686] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 1 LEADER]: Becoming Leader. State: Replica: d2dbcb067815446b9ee3ca818a024e66, State: Running, Role: LEADER
I20260810 04:45:55.292624 19041 consensus_queue.cc:237] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:55.293344 18175 catalog_manager.cc:5495] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1). New cstate: current_term: 1 leader_uuid: "d2dbcb067815446b9ee3ca818a024e66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } health_report { overall_health: UNKNOWN } } }
I20260810 04:45:55.314006 18916 raft_consensus.cc:1261] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Refusing update from remote peer d2dbcb067815446b9ee3ca818a024e66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:55.314016 18691 raft_consensus.cc:1261] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Refusing update from remote peer d2dbcb067815446b9ee3ca818a024e66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:55.314249 19041 consensus_queue.cc:1035] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:55.314394 19096 consensus_queue.cc:1035] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 04:45:55.318853 19075 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 04:45:56.842980 18175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:56.843092 18175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:56.843961 18175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: Sending DeleteTablet for 3 replicas of tablet 48f9c4343e934391aa28df42de2402b8
I20260810 04:45:56.844318 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f9c4343e934391aa28df42de2402b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:56 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:43054
I20260810 04:45:56.844321 18410 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f9c4343e934391aa28df42de2402b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:56 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:42338
I20260810 04:45:56.844525 18661 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f9c4343e934391aa28df42de2402b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:56 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:54338
I20260810 04:45:56.844543 19160 tablet_replica.cc:330] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: stopping tablet replica
I20260810 04:45:56.844609 19160 raft_consensus.cc:2229] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:56.844646 19160 raft_consensus.cc:2258] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:56.844743 19161 tablet_replica.cc:330] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: stopping tablet replica
I20260810 04:45:56.844789 19161 raft_consensus.cc:2229] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:45:56.844838 19161 raft_consensus.cc:2258] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:56.845031 19160 ts_tablet_manager.cc:1905] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:56.846944 19161 ts_tablet_manager.cc:1905] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:56.847342 19160 ts_tablet_manager.cc:1918] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:45:56.847420 19160 log.cc:1196] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/48f9c4343e934391aa28df42de2402b8
I20260810 04:45:56.847452 19163 tablet_replica.cc:330] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: stopping tablet replica
I20260810 04:45:56.847530 19163 raft_consensus.cc:2229] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:56.847579 19163 raft_consensus.cc:2258] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:56.847733 19160 ts_tablet_manager.cc:1939] T 48f9c4343e934391aa28df42de2402b8 P 7c75387a86284e6c9acf15e94887c6d7: Deleting consensus metadata
I20260810 04:45:56.847867 19163 ts_tablet_manager.cc:1905] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:56.848088 18151 catalog_manager.cc:4841] TS 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349): tablet 48f9c4343e934391aa28df42de2402b8 (table test-table [id=9f6b4aafae1749da96da6fb14939a549]) successfully deleted
I20260810 04:45:56.848788 19161 ts_tablet_manager.cc:1918] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:45:56.848878 19161 log.cc:1196] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/48f9c4343e934391aa28df42de2402b8
I20260810 04:45:56.849143 19161 ts_tablet_manager.cc:1939] T 48f9c4343e934391aa28df42de2402b8 P d2dbcb067815446b9ee3ca818a024e66: Deleting consensus metadata
I20260810 04:45:56.849465 18152 catalog_manager.cc:4841] TS d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995): tablet 48f9c4343e934391aa28df42de2402b8 (table test-table [id=9f6b4aafae1749da96da6fb14939a549]) successfully deleted
I20260810 04:45:56.849485 19163 ts_tablet_manager.cc:1918] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:45:56.849538 19163 log.cc:1196] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/48f9c4343e934391aa28df42de2402b8
I20260810 04:45:56.849736 19163 ts_tablet_manager.cc:1939] T 48f9c4343e934391aa28df42de2402b8 P eaf6501c861b45749957c81dac1ad999: Deleting consensus metadata
I20260810 04:45:56.850057 18151 catalog_manager.cc:4841] TS eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237): tablet 48f9c4343e934391aa28df42de2402b8 (table test-table [id=9f6b4aafae1749da96da6fb14939a549]) successfully deleted
Aug 10 04:45:56 dist-test-slave-8hhm krb5kdc[18093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337156, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:45:56 dist-test-slave-8hhm krb5kdc[18093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337156, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:45:56.876260 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:56.864627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.14.237.62:35585)
0810 04:45:56.864785 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:56.864787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:56.864798 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:45:56.864882 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:56.864883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:56.864895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:56.864981 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:45:56.865600 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.866083 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.866758 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.866944 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.866994 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:56.873570 (+ 6576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:45:56.873572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:45:56.873572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:45:56.873579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:45:56.875126 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:56.875586 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:56.875587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:56.875588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:56.875651 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:45:56.875814 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:45:56.875815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:45:56.875815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:45:56.875946 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:45:56.876023 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:56.876120 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:45:56.876183 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260810 04:45:56.877986 18175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:45:56.878134 18175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:56.879602 18410 tablet_service.cc:1467] Processing CreateTablet for tablet 167df5b38659425b8d3af4afd11f5648 (DEFAULT_TABLE table=test-table [id=5bd5f3d7f26a4e9981f802f14bdf1c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:56.879640 18661 tablet_service.cc:1467] Processing CreateTablet for tablet 167df5b38659425b8d3af4afd11f5648 (DEFAULT_TABLE table=test-table [id=5bd5f3d7f26a4e9981f802f14bdf1c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:56.879647 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 167df5b38659425b8d3af4afd11f5648 (DEFAULT_TABLE table=test-table [id=5bd5f3d7f26a4e9981f802f14bdf1c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:56.879781 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167df5b38659425b8d3af4afd11f5648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:56.879781 18886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167df5b38659425b8d3af4afd11f5648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:56.879799 18410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167df5b38659425b8d3af4afd11f5648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:56.881326 19177 tablet_bootstrap.cc:492] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: Bootstrap starting.
I20260810 04:45:56.881681 19177 tablet_bootstrap.cc:654] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:56.882014 19176 tablet_bootstrap.cc:492] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: Bootstrap starting.
I20260810 04:45:56.882354 19176 tablet_bootstrap.cc:654] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:56.882789 19177 tablet_bootstrap.cc:492] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: No bootstrap required, opened a new log
I20260810 04:45:56.882845 19177 ts_tablet_manager.cc:1397] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:56.882990 19176 tablet_bootstrap.cc:492] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: No bootstrap required, opened a new log
I20260810 04:45:56.883071 19176 ts_tablet_manager.cc:1397] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:56.883030 19177 raft_consensus.cc:348] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.883134 19177 raft_consensus.cc:374] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:56.883162 19177 raft_consensus.cc:729] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c75387a86284e6c9acf15e94887c6d7, State: Initialized, Role: FOLLOWER
I20260810 04:45:56.883239 19177 consensus_queue.cc:260] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.883277 19176 raft_consensus.cc:348] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.883366 19176 raft_consensus.cc:374] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:56.883396 19176 raft_consensus.cc:729] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf6501c861b45749957c81dac1ad999, State: Initialized, Role: FOLLOWER
I20260810 04:45:56.883428 19177 ts_tablet_manager.cc:1428] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:56.883476 19176 consensus_queue.cc:260] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.883669 19176 ts_tablet_manager.cc:1428] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:56.885367 19174 tablet_bootstrap.cc:492] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: Bootstrap starting.
I20260810 04:45:56.885771 19174 tablet_bootstrap.cc:654] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:56.886835 19174 tablet_bootstrap.cc:492] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: No bootstrap required, opened a new log
I20260810 04:45:56.886890 19174 ts_tablet_manager.cc:1397] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:56.887023 19174 raft_consensus.cc:348] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.887104 19174 raft_consensus.cc:374] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:56.887127 19174 raft_consensus.cc:729] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2dbcb067815446b9ee3ca818a024e66, State: Initialized, Role: FOLLOWER
I20260810 04:45:56.887190 19174 consensus_queue.cc:260] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.887329 19174 ts_tablet_manager.cc:1428] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:56.975997 19183 raft_consensus.cc:482] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:56.976114 19183 raft_consensus.cc:504] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.976553 19183 leader_election.cc:288] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995), 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:56.979871 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:56.976726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:46349 (local address 127.14.237.2:48817)
0810 04:45:56.976916 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:56.976927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:56.976977 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:56.977322 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:56.977323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:56.977326 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:56.977420 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:56.977422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.978090 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.978090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:56.979070 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:56.979073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.979669 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.979670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:56.979731 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:56.979734 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:45:56.979779 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 04:45:56.979809 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":117,"threads_started":1}
I20260810 04:45:56.979931 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:56.976918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:48817 (local address 127.14.237.3:46349)
0810 04:45:56.977128 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:56.977130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:56.977136 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:45:56.977157 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:56.977157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:56.977185 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:56.977314 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:56.977515 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.978013 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.979161 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.979588 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.979663 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:56.979672 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:56.979806 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 04:45:56.979848 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20260810 04:45:56.980357 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167df5b38659425b8d3af4afd11f5648" candidate_uuid: "eaf6501c861b45749957c81dac1ad999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75387a86284e6c9acf15e94887c6d7" is_pre_election: true
I20260810 04:45:56.980437 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:56.976735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41995 (local address 127.14.237.2:56175)
0810 04:45:56.977018 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:56.977030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:56.977093 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:56.977490 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:56.977491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:56.977492 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:56.977879 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:56.977880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.978508 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.978509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:56.979423 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:56.979425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.980173 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.980174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:56.980254 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:56.980258 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:45:56.980318 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:45:56.980364 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260810 04:45:56.980444 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:56.977072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:56175 (local address 127.14.237.1:41995)
0810 04:45:56.977291 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:56.977293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:56.977332 (+ 39us) server_negotiation.cc:407] Connection header received
0810 04:45:56.977371 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:56.977372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:56.977384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:56.977470 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:56.977960 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.978426 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.979500 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:56.980087 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:56.980164 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:56.980174 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:56.980341 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 04:45:56.980382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
I20260810 04:45:56.980453 18916 raft_consensus.cc:2454] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf6501c861b45749957c81dac1ad999 in term 0.
I20260810 04:45:56.980757 18453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167df5b38659425b8d3af4afd11f5648" candidate_uuid: "eaf6501c861b45749957c81dac1ad999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2dbcb067815446b9ee3ca818a024e66" is_pre_election: true
I20260810 04:45:56.980829 18579 leader_election.cc:302] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c75387a86284e6c9acf15e94887c6d7, eaf6501c861b45749957c81dac1ad999; no voters:
I20260810 04:45:56.980904 18453 raft_consensus.cc:2454] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf6501c861b45749957c81dac1ad999 in term 0.
I20260810 04:45:56.980976 19183 raft_consensus.cc:2781] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:56.981017 19183 raft_consensus.cc:482] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:56.981060 19183 raft_consensus.cc:3037] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:56.981827 19183 raft_consensus.cc:504] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.981984 19183 leader_election.cc:288] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [CANDIDATE]: Term 1 election: Requested vote from peers d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995), 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349)
I20260810 04:45:56.982123 18453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167df5b38659425b8d3af4afd11f5648" candidate_uuid: "eaf6501c861b45749957c81dac1ad999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2dbcb067815446b9ee3ca818a024e66"
I20260810 04:45:56.982197 18453 raft_consensus.cc:3037] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:56.982156 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167df5b38659425b8d3af4afd11f5648" candidate_uuid: "eaf6501c861b45749957c81dac1ad999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75387a86284e6c9acf15e94887c6d7"
I20260810 04:45:56.982250 18916 raft_consensus.cc:3037] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:56.982935 18916 raft_consensus.cc:2454] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf6501c861b45749957c81dac1ad999 in term 1.
I20260810 04:45:56.982930 18453 raft_consensus.cc:2454] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf6501c861b45749957c81dac1ad999 in term 1.
I20260810 04:45:56.983127 18580 leader_election.cc:302] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2dbcb067815446b9ee3ca818a024e66, eaf6501c861b45749957c81dac1ad999; no voters:
I20260810 04:45:56.983213 19183 raft_consensus.cc:2781] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:56.983368 19183 raft_consensus.cc:686] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 1 LEADER]: Becoming Leader. State: Replica: eaf6501c861b45749957c81dac1ad999, State: Running, Role: LEADER
I20260810 04:45:56.983459 19183 consensus_queue.cc:237] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:56.984158 18176 catalog_manager.cc:5495] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf6501c861b45749957c81dac1ad999 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "eaf6501c861b45749957c81dac1ad999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } health_report { overall_health: UNKNOWN } } }
I20260810 04:45:57.005456 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.002813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.14.237.2:40237)
0810 04:45:57.002961 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:57.002963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:57.002971 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:45:57.003005 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:57.003005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:57.003021 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:57.003104 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:57.003279 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.003766 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.004530 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.004790 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.004871 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.005053 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.005168 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.005274 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:57.005339 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 04:45:57.005359 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260810 04:45:57.010336 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.007825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:47695 (local address 127.14.237.1:41995)
0810 04:45:57.007891 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:57.007893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:57.008017 (+ 124us) server_negotiation.cc:407] Connection header received
0810 04:45:57.008096 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:57.008097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:57.008109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:57.008187 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:57.008328 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.008736 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.009626 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.009828 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.009902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.010078 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.010112 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.010172 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:57.010240 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 04:45:57.010261 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:45:57.010371 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.007741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41995 (local address 127.14.237.2:47695)
0810 04:45:57.007947 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:57.007957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:57.008032 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:57.008190 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:57.008191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:57.008193 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:45:57.008258 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.008259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.008812 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.008813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.009546 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.009549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.009902 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.009903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.009965 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.009986 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.010175 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.010177 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:45:57.010235 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:45:57.010275 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260810 04:45:57.012358 18916 raft_consensus.cc:1261] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Refusing update from remote peer eaf6501c861b45749957c81dac1ad999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:57.012413 18453 raft_consensus.cc:1261] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Refusing update from remote peer eaf6501c861b45749957c81dac1ad999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:57.012660 19198 consensus_queue.cc:1035] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:57.012830 19183 consensus_queue.cc:1035] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:57.019865 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.017855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:53407 (local address 127.14.237.2:40237)
0810 04:45:57.017904 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:57.017906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:57.018000 (+ 94us) server_negotiation.cc:407] Connection header received
0810 04:45:57.018052 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:57.018052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:57.018061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:57.018103 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:57.018258 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.018642 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.019262 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.019411 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.019494 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.019644 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.019671 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.019724 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:57.019795 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:45:57.019811 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260810 04:45:57.019904 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.017801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40237 (local address 127.14.237.1:53407)
0810 04:45:57.017945 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:57.017954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:57.018005 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:57.018141 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:57.018142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:57.018143 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:45:57.018179 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.018180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.018706 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.018707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.019181 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.019182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.019497 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.019498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.019558 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.019562 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.019730 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.019731 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:45:57.019791 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:45:57.019831 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":107,"threads_started":1}
I20260810 04:45:57.026748 18176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:57.026850 18176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:57.027493 18176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: Sending DeleteTablet for 3 replicas of tablet 167df5b38659425b8d3af4afd11f5648
I20260810 04:45:57.027748 18661 tablet_service.cc:1514] Processing DeleteTablet for tablet 167df5b38659425b8d3af4afd11f5648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:57 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:54338
I20260810 04:45:57.027763 18410 tablet_service.cc:1514] Processing DeleteTablet for tablet 167df5b38659425b8d3af4afd11f5648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:57 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:42338
I20260810 04:45:57.027838 19163 tablet_replica.cc:330] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: stopping tablet replica
I20260810 04:45:57.027838 19161 tablet_replica.cc:330] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: stopping tablet replica
I20260810 04:45:57.027829 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 167df5b38659425b8d3af4afd11f5648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:57 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:43054
I20260810 04:45:57.027881 19161 raft_consensus.cc:2229] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:57.027881 19163 raft_consensus.cc:2229] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:45:57.027915 19161 raft_consensus.cc:2258] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:57.027916 19160 tablet_replica.cc:330] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: stopping tablet replica
I20260810 04:45:57.027957 19163 raft_consensus.cc:2258] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:57.027957 19160 raft_consensus.cc:2229] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:57.028007 19160 raft_consensus.cc:2258] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:57.028223 19161 ts_tablet_manager.cc:1905] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:57.028405 19163 ts_tablet_manager.cc:1905] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:57.028374 18176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:45:57.028506 18176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:45:57.029083 19160 ts_tablet_manager.cc:1905] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:57.030272 19163 ts_tablet_manager.cc:1918] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:57.030347 19163 log.cc:1196] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/167df5b38659425b8d3af4afd11f5648
I20260810 04:45:57.030560 19163 ts_tablet_manager.cc:1939] T 167df5b38659425b8d3af4afd11f5648 P eaf6501c861b45749957c81dac1ad999: Deleting consensus metadata
I20260810 04:45:57.030646 19160 ts_tablet_manager.cc:1918] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:57.030709 19160 log.cc:1196] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/167df5b38659425b8d3af4afd11f5648
I20260810 04:45:57.030861 18151 catalog_manager.cc:4841] TS eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237): tablet 167df5b38659425b8d3af4afd11f5648 (table test-table [id=5bd5f3d7f26a4e9981f802f14bdf1c1a]) successfully deleted
I20260810 04:45:57.030921 19161 ts_tablet_manager.cc:1918] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:45:57.030956 19160 ts_tablet_manager.cc:1939] T 167df5b38659425b8d3af4afd11f5648 P 7c75387a86284e6c9acf15e94887c6d7: Deleting consensus metadata
I20260810 04:45:57.030972 19161 log.cc:1196] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/167df5b38659425b8d3af4afd11f5648
I20260810 04:45:57.031180 19161 ts_tablet_manager.cc:1939] T 167df5b38659425b8d3af4afd11f5648 P d2dbcb067815446b9ee3ca818a024e66: Deleting consensus metadata
I20260810 04:45:57.031481 18152 catalog_manager.cc:4841] TS d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995): tablet 167df5b38659425b8d3af4afd11f5648 (table test-table [id=5bd5f3d7f26a4e9981f802f14bdf1c1a]) successfully deleted
I20260810 04:45:57.031713 18410 tablet_service.cc:1467] Processing CreateTablet for tablet 6be3daac65844fadac9583c5aa366922 (DEFAULT_TABLE table=test-table [id=00e53128e526444a857eb21dbeca8151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:57.031704 18661 tablet_service.cc:1467] Processing CreateTablet for tablet 6be3daac65844fadac9583c5aa366922 (DEFAULT_TABLE table=test-table [id=00e53128e526444a857eb21dbeca8151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:57.031713 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 6be3daac65844fadac9583c5aa366922 (DEFAULT_TABLE table=test-table [id=00e53128e526444a857eb21dbeca8151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:45:57.031822 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be3daac65844fadac9583c5aa366922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:57.031822 18410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be3daac65844fadac9583c5aa366922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:57.031845 18886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6be3daac65844fadac9583c5aa366922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:45:57.032974 19174 tablet_bootstrap.cc:492] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66: Bootstrap starting.
I20260810 04:45:57.033002 19176 tablet_bootstrap.cc:492] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: Bootstrap starting.
I20260810 04:45:57.033313 19174 tablet_bootstrap.cc:654] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:57.033316 19177 tablet_bootstrap.cc:492] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: Bootstrap starting.
I20260810 04:45:57.033427 19176 tablet_bootstrap.cc:654] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:57.033885 19177 tablet_bootstrap.cc:654] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: Neither blocks nor log segments found. Creating new log.
I20260810 04:45:57.034196 18151 catalog_manager.cc:4841] TS 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349): tablet 167df5b38659425b8d3af4afd11f5648 (table test-table [id=5bd5f3d7f26a4e9981f802f14bdf1c1a]) successfully deleted
I20260810 04:45:57.034240 19174 tablet_bootstrap.cc:492] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66: No bootstrap required, opened a new log
I20260810 04:45:57.034281 19174 ts_tablet_manager.cc:1397] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:45:57.034399 19176 tablet_bootstrap.cc:492] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: No bootstrap required, opened a new log
I20260810 04:45:57.034394 19174 raft_consensus.cc:348] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.034452 19176 ts_tablet_manager.cc:1397] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:45:57.034461 19174 raft_consensus.cc:374] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:57.034484 19174 raft_consensus.cc:729] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2dbcb067815446b9ee3ca818a024e66, State: Initialized, Role: FOLLOWER
I20260810 04:45:57.034529 19174 consensus_queue.cc:260] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.034618 19174 ts_tablet_manager.cc:1428] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:45:57.034610 19176 raft_consensus.cc:348] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.034672 19176 raft_consensus.cc:374] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:57.034700 19176 raft_consensus.cc:729] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf6501c861b45749957c81dac1ad999, State: Initialized, Role: FOLLOWER
I20260810 04:45:57.034752 19176 consensus_queue.cc:260] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.034880 19176 ts_tablet_manager.cc:1428] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:45:57.035588 19177 tablet_bootstrap.cc:492] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: No bootstrap required, opened a new log
I20260810 04:45:57.035656 19177 ts_tablet_manager.cc:1397] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:45:57.035795 19177 raft_consensus.cc:348] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.035869 19177 raft_consensus.cc:374] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:45:57.035893 19177 raft_consensus.cc:729] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c75387a86284e6c9acf15e94887c6d7, State: Initialized, Role: FOLLOWER
I20260810 04:45:57.035939 19177 consensus_queue.cc:260] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.036020 19177 ts_tablet_manager.cc:1428] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:45:57.062520 19182 raft_consensus.cc:482] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:45:57.062616 19182 raft_consensus.cc:504] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.063036 19182 leader_election.cc:288] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995), eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237)
I20260810 04:45:57.066746 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.063240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40237 (local address 127.14.237.3:35749)
0810 04:45:57.063572 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:57.063583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:57.063663 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:57.063843 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:57.063844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:57.063846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:57.063928 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.063929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.064867 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.064868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.065930 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.065931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.066472 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.066473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.066543 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.066549 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:45:57.066615 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:45:57.066656 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":118,"threads_started":1}
I20260810 04:45:57.066735 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.063341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:35749 (local address 127.14.237.2:40237)
0810 04:45:57.063390 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:57.063392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:57.063634 (+ 242us) server_negotiation.cc:407] Connection header received
0810 04:45:57.063748 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:57.063750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:57.063759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:57.063836 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:57.064013 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.064763 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.066031 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.066388 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.066473 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.066483 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:57.066632 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:45:57.066670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:45:57.066968 18691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be3daac65844fadac9583c5aa366922" candidate_uuid: "7c75387a86284e6c9acf15e94887c6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf6501c861b45749957c81dac1ad999" is_pre_election: true
I20260810 04:45:57.067081 18691 raft_consensus.cc:2454] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c75387a86284e6c9acf15e94887c6d7 in term 0.
I20260810 04:45:57.067268 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.063435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:44977 (local address 127.14.237.1:41995)
0810 04:45:57.063483 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:57.063485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:57.063560 (+ 75us) server_negotiation.cc:407] Connection header received
0810 04:45:57.063673 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:57.063674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:57.063685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:57.063714 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:45:57.063931 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.064510 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.066014 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.066571 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.066645 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.066653 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:57.067151 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 04:45:57.067199 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:45:57.067270 18800 leader_election.cc:302] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c75387a86284e6c9acf15e94887c6d7, eaf6501c861b45749957c81dac1ad999; no voters:
I20260810 04:45:57.067412 19182 raft_consensus.cc:2781] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:45:57.067464 19182 raft_consensus.cc:482] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:45:57.067487 19182 raft_consensus.cc:3037] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:57.067555 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.063195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41995 (local address 127.14.237.3:44977)
0810 04:45:57.063492 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:57.063504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:57.063591 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:57.063794 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:57.063795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:57.063797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:45:57.063869 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.063871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.064592 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.064593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.065912 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.065914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.066689 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.066690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.066778 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.066782 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:45:57.067350 (+ 568us) client_negotiation.cc:240] Negotiation successful
0810 04:45:57.067392 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":46,"thread_start_us":77,"threads_started":1}
I20260810 04:45:57.067793 18453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be3daac65844fadac9583c5aa366922" candidate_uuid: "7c75387a86284e6c9acf15e94887c6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2dbcb067815446b9ee3ca818a024e66" is_pre_election: true
I20260810 04:45:57.067884 18453 raft_consensus.cc:2454] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c75387a86284e6c9acf15e94887c6d7 in term 0.
I20260810 04:45:57.068346 19182 raft_consensus.cc:504] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.068496 19182 leader_election.cc:288] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [CANDIDATE]: Term 1 election: Requested vote from peers d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995), eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237)
I20260810 04:45:57.068681 18453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be3daac65844fadac9583c5aa366922" candidate_uuid: "7c75387a86284e6c9acf15e94887c6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2dbcb067815446b9ee3ca818a024e66"
I20260810 04:45:57.068681 18691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6be3daac65844fadac9583c5aa366922" candidate_uuid: "7c75387a86284e6c9acf15e94887c6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf6501c861b45749957c81dac1ad999"
I20260810 04:45:57.068789 18453 raft_consensus.cc:3037] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:57.068790 18691 raft_consensus.cc:3037] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:45:57.069490 18453 raft_consensus.cc:2454] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c75387a86284e6c9acf15e94887c6d7 in term 1.
I20260810 04:45:57.069621 18691 raft_consensus.cc:2454] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c75387a86284e6c9acf15e94887c6d7 in term 1.
I20260810 04:45:57.069656 18801 leader_election.cc:302] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c75387a86284e6c9acf15e94887c6d7, d2dbcb067815446b9ee3ca818a024e66; no voters:
I20260810 04:45:57.069752 19182 raft_consensus.cc:2781] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:45:57.069911 19182 raft_consensus.cc:686] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7c75387a86284e6c9acf15e94887c6d7, State: Running, Role: LEADER
I20260810 04:45:57.070008 19182 consensus_queue.cc:237] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } }
I20260810 04:45:57.070710 18176 catalog_manager.cc:5495] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3). New cstate: current_term: 1 leader_uuid: "7c75387a86284e6c9acf15e94887c6d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c75387a86284e6c9acf15e94887c6d7" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46349 } health_report { overall_health: HEALTHY } } }
I20260810 04:45:57.087167 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.084748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.14.237.3:46349)
0810 04:45:57.084792 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:57.084794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:57.084802 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:45:57.084840 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:57.084840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:57.084862 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:57.084947 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:57.085075 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.085615 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.086416 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.086622 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.086693 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.086831 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.086941 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.087021 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:57.087063 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 04:45:57.087082 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:45:57.092181 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.089735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:38731 (local address 127.14.237.1:41995)
0810 04:45:57.089773 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:57.089775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:57.089835 (+ 60us) server_negotiation.cc:407] Connection header received
0810 04:45:57.089904 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:57.089905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:57.089913 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:57.089956 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:57.090078 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.090491 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.091503 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.091692 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.091749 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.091973 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.091997 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.092034 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:57.092116 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 04:45:57.092134 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:45:57.092247 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.089600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41995 (local address 127.14.237.3:38731)
0810 04:45:57.089779 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:57.089790 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:57.089850 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:57.089962 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:57.089963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:57.089964 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:45:57.090022 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.090023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.090575 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.090576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.091420 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.091423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.091789 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.091790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.091870 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.091886 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.092059 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.092061 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:45:57.092115 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:45:57.092158 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260810 04:45:57.094252 18453 raft_consensus.cc:1261] T 6be3daac65844fadac9583c5aa366922 P d2dbcb067815446b9ee3ca818a024e66 [term 1 FOLLOWER]: Refusing update from remote peer 7c75387a86284e6c9acf15e94887c6d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:57.094312 18691 raft_consensus.cc:1261] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Refusing update from remote peer 7c75387a86284e6c9acf15e94887c6d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:45:57.094496 19182 consensus_queue.cc:1035] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dbcb067815446b9ee3ca818a024e66" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:57.094650 19225 consensus_queue.cc:1035] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf6501c861b45749957c81dac1ad999" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:45:57.101300 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.098900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:46349 (local address 127.14.237.1:37537)
0810 04:45:57.098949 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:57.098959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:45:57.098993 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:45:57.099129 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:45:57.099130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:45:57.099131 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:45:57.099169 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.099171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.099731 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.099732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.100492 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:45:57.100495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.100845 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.100846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:45:57.100924 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.100932 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.101102 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.101105 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:45:57.101158 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:45:57.101232 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:45:57.101336 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:57.098966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:37537 (local address 127.14.237.3:46349)
0810 04:45:57.099007 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:45:57.099009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:45:57.099017 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:45:57.099049 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:45:57.099049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:45:57.099057 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:45:57.099105 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:45:57.099234 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.099645 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.100592 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:45:57.100757 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:45:57.100799 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:45:57.101024 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:45:57.101053 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:45:57.101099 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:45:57.101256 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 04:45:57.101276 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:45:57.101784 19129 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 04:45:58.620883 18176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:58.620986 18176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:45:58.621934 18176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb75b362746a4314bf9919495be369bc: Sending DeleteTablet for 3 replicas of tablet 6be3daac65844fadac9583c5aa366922
I20260810 04:45:58.622282 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be3daac65844fadac9583c5aa366922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:58 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:43054
I20260810 04:45:58.622289 18661 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be3daac65844fadac9583c5aa366922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:58 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:54338
I20260810 04:45:58.622390 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18301
I20260810 04:45:58.622483 18411 tablet_service.cc:1514] Processing DeleteTablet for tablet 6be3daac65844fadac9583c5aa366922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:45:58 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:42338
I20260810 04:45:58.631388 19281 tablet_replica.cc:330] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: stopping tablet replica
I20260810 04:45:58.631448 19281 raft_consensus.cc:2229] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:45:58.631477 19281 raft_consensus.cc:2258] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:58.631600 19280 tablet_replica.cc:330] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: stopping tablet replica
I20260810 04:45:58.631668 19280 raft_consensus.cc:2229] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:45:58.631783 19280 raft_consensus.cc:2258] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:45:58.631932 19281 ts_tablet_manager.cc:1905] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:58.632095 19280 ts_tablet_manager.cc:1905] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:45:58.634023 19281 ts_tablet_manager.cc:1918] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:45:58.634023 19280 ts_tablet_manager.cc:1918] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:45:58.634089 19281 log.cc:1196] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/6be3daac65844fadac9583c5aa366922
I20260810 04:45:58.634128 19280 log.cc:1196] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/6be3daac65844fadac9583c5aa366922
I20260810 04:45:58.634323 19281 ts_tablet_manager.cc:1939] T 6be3daac65844fadac9583c5aa366922 P eaf6501c861b45749957c81dac1ad999: Deleting consensus metadata
I20260810 04:45:58.634333 19280 ts_tablet_manager.cc:1939] T 6be3daac65844fadac9583c5aa366922 P 7c75387a86284e6c9acf15e94887c6d7: Deleting consensus metadata
I20260810 04:45:58.634636 18151 catalog_manager.cc:4841] TS eaf6501c861b45749957c81dac1ad999 (127.14.237.2:40237): tablet 6be3daac65844fadac9583c5aa366922 (table test-table [id=00e53128e526444a857eb21dbeca8151]) successfully deleted
I20260810 04:45:58.634712 18151 catalog_manager.cc:4841] TS 7c75387a86284e6c9acf15e94887c6d7 (127.14.237.3:46349): tablet 6be3daac65844fadac9583c5aa366922 (table test-table [id=00e53128e526444a857eb21dbeca8151]) successfully deleted
W20260810 04:45:58.641637 18152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:41995 (error 108)
I20260810 04:45:58.641888 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18553
I20260810 04:45:58.643554 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:45:58.642071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41995 (local address 127.0.0.1:42368)
0810 04:45:58.643432 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0810 04:45:58.643486 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:41995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1336,"thread_start_us":68,"threads_started":1}
W20260810 04:45:58.643663 18152 catalog_manager.cc:4584] TS d2dbcb067815446b9ee3ca818a024e66 (127.14.237.1:41995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6be3daac65844fadac9583c5aa366922: Network error: Client connection negotiation failed: client connection to 127.14.237.1:41995: connect: Connection refused (error 111)
I20260810 04:45:58.650761 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18774
I20260810 04:45:58.658686 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18117
2026-08-10T04:45:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6202 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:45:58 dist-test-slave-8hhm krb5kdc[19289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:45:58 dist-test-slave-8hhm krb5kdc[19289](info): set up 2 sockets
Aug 10 04:45:58 dist-test-slave-8hhm krb5kdc[19289](info): commencing operation
krb5kdc: starting...
W20260810 04:45:59.923549 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:00.041601 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:34777
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:00.124634 19312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:00.124864 19312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:00.124914 19312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:00.126196 19312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:00.126263 19312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:00.126288 19312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:00.126319 19312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:00.126343 19312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:00.127789 19312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:34777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:00.128067 19312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:00.128293 19312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:00.133399 19321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:00.133360 19319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:00.133365 19312 server_base.cc:1034] running on GCE node
W20260810 04:46:00.133399 19317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:00.133862 19312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:00.134176 19312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:00.135347 19312 hybrid_clock.cc:648] HybridClock initialized: now 1786337160135334 us; error 45 us; skew 500 ppm
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:00.137871 19312 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:00.138302 19312 webserver.cc:460] Webserver started at http://127.14.237.62:43961/ using document root <none> and password file <none>
I20260810 04:46:00.138551 19312 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:00.138638 19312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:00.138792 19312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:00.139824 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "55b9f6a119474e598e13e4b35239796a"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "b6ce199139f8377fb49d8a668dfcfde3"
server_key_iv: "469a176d85f94826e77bdffa790c225c"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.140195 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "55b9f6a119474e598e13e4b35239796a"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "b6ce199139f8377fb49d8a668dfcfde3"
server_key_iv: "469a176d85f94826e77bdffa790c225c"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.141539 19312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:00.142295 19334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.142473 19312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:00.142581 19312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "55b9f6a119474e598e13e4b35239796a"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "b6ce199139f8377fb49d8a668dfcfde3"
server_key_iv: "469a176d85f94826e77bdffa790c225c"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.142659 19312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:00.152143 19312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:00.154096 19312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:00.154254 19312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:00.158423 19312 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:34777
I20260810 04:46:00.158473 19435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:34777 every 8 connection(s)
I20260810 04:46:00.158833 19312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:00.159762 19436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:00.162098 19436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a: Bootstrap starting.
I20260810 04:46:00.162675 19436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:00.162973 19436 log.cc:824] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:00.163681 19436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a: No bootstrap required, opened a new log
I20260810 04:46:00.165201 19436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b9f6a119474e598e13e4b35239796a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34777 } }
I20260810 04:46:00.165361 19436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:00.165382 19436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b9f6a119474e598e13e4b35239796a, State: Initialized, Role: FOLLOWER
I20260810 04:46:00.165551 19436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b9f6a119474e598e13e4b35239796a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34777 } }
I20260810 04:46:00.165616 19436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:00.165660 19436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:00.165709 19436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:00.166221 19436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b9f6a119474e598e13e4b35239796a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34777 } }
I20260810 04:46:00.166307 19436 leader_election.cc:302] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55b9f6a119474e598e13e4b35239796a; no voters:
I20260810 04:46:00.166538 19436 leader_election.cc:288] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:00.166646 19440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:00.166778 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19312
I20260810 04:46:00.166833 19440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [term 1 LEADER]: Becoming Leader. State: Replica: 55b9f6a119474e598e13e4b35239796a, State: Running, Role: LEADER
I20260810 04:46:00.166841 19436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:00.166891 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:00.166934 19440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b9f6a119474e598e13e4b35239796a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34777 } }
I20260810 04:46:00.167073 15284 external_mini_cluster.cc:1428] Setting key 9ce433bb13d21d559eb7a04ca7d6d7c9
I20260810 04:46:00.167362 19441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55b9f6a119474e598e13e4b35239796a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b9f6a119474e598e13e4b35239796a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34777 } } }
I20260810 04:46:00.167448 19441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:00.167599 19442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b9f6a119474e598e13e4b35239796a. Latest consensus state: current_term: 1 leader_uuid: "55b9f6a119474e598e13e4b35239796a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b9f6a119474e598e13e4b35239796a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34777 } } }
I20260810 04:46:00.167655 19442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:00.168179 19445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:00.168764 19445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:00.170388 19445 catalog_manager.cc:1347] Generated new cluster ID: 91e95497846a43bc9ac402a1707d8dae
I20260810 04:46:00.170440 19445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:00.178707 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:00.167742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.14.237.62:34777)
0810 04:46:00.168052 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:00.168062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:00.168097 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:46:00.168208 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:00.168213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:00.168288 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:00.168441 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:00.169246 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.170598 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.171660 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.172338 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.172451 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:00.175219 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:00.175223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:00.175223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:00.175235 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:00.177109 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:00.177523 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:00.177525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:00.177525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:00.177641 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:00.177858 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:00.177859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:00.177859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:00.178115 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:00.178249 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:00.178379 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 04:46:00.178498 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":94,"threads_started":1}
I20260810 04:46:00.192500 19445 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:00.193197 19445 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:00.204708 19445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55b9f6a119474e598e13e4b35239796a: Generated new TSK 0
I20260810 04:46:00.204893 19445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:46:00.235735 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34777
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:00.336362 19475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:00.336557 19475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:00.336585 19475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:00.337863 19475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:00.337903 19475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:00.337929 19475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:00.339314 19475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:00.339600 19475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:00.339816 19475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:00.344895 19487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:00.344944 19484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:00.344946 19483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:00.345151 19475 server_base.cc:1034] running on GCE node
I20260810 04:46:00.345372 19475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:00.345561 19475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:00.346756 19475 hybrid_clock.cc:648] HybridClock initialized: now 1786337160346733 us; error 34 us; skew 500 ppm
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:00.349141 19475 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:46:00.349663 19475 webserver.cc:460] Webserver started at http://127.14.237.1:39473/ using document root <none> and password file <none>
I20260810 04:46:00.349890 19475 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:00.349960 19475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:00.350113 19475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:00.351197 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "829c55c8fc3949478a705f4bb7f68925"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "06bbde2dc0877d5b05b3e5dd9c916160"
server_key_iv: "8c76cd1ad5b561f5e6565986a3d60815"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.351553 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "829c55c8fc3949478a705f4bb7f68925"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "06bbde2dc0877d5b05b3e5dd9c916160"
server_key_iv: "8c76cd1ad5b561f5e6565986a3d60815"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.352910 19475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:00.353711 19494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.353886 19475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.353989 19475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "829c55c8fc3949478a705f4bb7f68925"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "06bbde2dc0877d5b05b3e5dd9c916160"
server_key_iv: "8c76cd1ad5b561f5e6565986a3d60815"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.354069 19475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:00.367205 19475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:00.369202 19475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:00.369377 19475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:00.369603 19475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:00.369931 19475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:00.369985 19475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.370026 19475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:00.370100 19475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.375976 19475 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:41865
I20260810 04:46:00.376027 19678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:41865 every 8 connection(s)
I20260810 04:46:00.376322 19475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:00.381491 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19475
I20260810 04:46:00.381567 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:00.381754 15284 external_mini_cluster.cc:1428] Setting key 2c91f407eaad57712f99cff7b6bb4b4a
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:00.385526 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:00.377010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:55555 (local address 127.14.237.62:34777)
0810 04:46:00.377072 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:00.377074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:00.377385 (+ 311us) server_negotiation.cc:407] Connection header received
0810 04:46:00.377837 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:00.377838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:00.377849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:00.377912 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:00.378784 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.379262 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.380178 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.380356 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.380404 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:00.382764 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:00.382766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:00.382766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:00.382773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:00.384074 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:00.384473 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:00.384474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:00.384474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:00.384539 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:00.384795 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:00.384796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:00.384797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:00.384942 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:00.385018 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:00.385345 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 04:46:00.385439 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:46:00.385694 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:00.377005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34777 (local address 127.14.237.1:55555)
0810 04:46:00.377190 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:00.377261 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:00.377751 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:00.377963 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:00.377966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:00.378519 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:00.378703 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:00.378708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.379346 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.379347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:00.380093 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:00.380095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.380428 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.380429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:00.380524 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:00.380956 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:00.380964 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:00.382660 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:00.384166 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:00.384173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:00.384175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:00.384393 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:00.384614 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:00.384614 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:00.384615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:00.384728 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:00.385020 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:00.385028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:00.385169 (+ 141us) client_negotiation.cc:769] Sending connection context
0810 04:46:00.385306 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 04:46:00.385463 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260810 04:46:00.386271 19681 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34777
I20260810 04:46:00.386373 19681 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:00.386574 19681 heartbeater.cc:511] Master 127.14.237.62:34777 requested a full tablet report, sending...
I20260810 04:46:00.387138 19375 ts_manager.cc:194] Registered new tserver with Master: 829c55c8fc3949478a705f4bb7f68925 (127.14.237.1:41865)
I20260810 04:46:00.387722 19375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:55555
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:46:00.408269 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34777
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:00.488936 19687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:00.489133 19687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:00.489156 19687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:00.490315 19687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:00.490386 19687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:00.490411 19687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:00.492004 19687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:00.492187 19687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:00.492367 19687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:46:00.497424 19687 server_base.cc:1034] running on GCE node
W20260810 04:46:00.497365 19693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:00.497365 19696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:00.497397 19694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:00.497768 19687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:00.497985 19687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:00.499185 19687 hybrid_clock.cc:648] HybridClock initialized: now 1786337160499170 us; error 25 us; skew 500 ppm
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:00.501715 19687 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:46:00.502174 19687 webserver.cc:460] Webserver started at http://127.14.237.2:42121/ using document root <none> and password file <none>
I20260810 04:46:00.502417 19687 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:00.502487 19687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:00.502648 19687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:00.503738 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "bf6bd32a6ee246e793150c9d1b9c9733"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "547084d5a712c0aa11eb40447b51e27b"
server_key_iv: "1819a1a28ce7ce8f20b006963197cd10"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.504120 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "bf6bd32a6ee246e793150c9d1b9c9733"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "547084d5a712c0aa11eb40447b51e27b"
server_key_iv: "1819a1a28ce7ce8f20b006963197cd10"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.505519 19687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:00.506259 19705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.506426 19687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:00.506533 19687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "bf6bd32a6ee246e793150c9d1b9c9733"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "547084d5a712c0aa11eb40447b51e27b"
server_key_iv: "1819a1a28ce7ce8f20b006963197cd10"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.506606 19687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:00.562367 19687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:00.564352 19687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:00.564522 19687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:00.564787 19687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:00.565111 19687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:00.565166 19687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.565201 19687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:00.565280 19687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.571409 19687 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:43095
I20260810 04:46:00.571462 19900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:43095 every 8 connection(s)
I20260810 04:46:00.571789 19687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:46:00.573660 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19687
I20260810 04:46:00.573746 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:00.573928 15284 external_mini_cluster.cc:1428] Setting key 7e5aaeff8d38ea803bc16a6e517bc851
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:00.581835 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:00.572504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:43593 (local address 127.14.237.62:34777)
0810 04:46:00.572547 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:00.572549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:00.572809 (+ 260us) server_negotiation.cc:407] Connection header received
0810 04:46:00.573441 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:00.573443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:00.573456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:00.573510 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:00.574698 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.575178 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.576162 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.576354 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.576430 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:00.578887 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:00.578889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:00.578889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:00.578897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:00.580348 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:00.580782 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:00.580783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:00.580784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:00.580847 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:00.581099 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:00.581100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:00.581101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:00.581278 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:00.581343 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:00.581634 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 04:46:00.581746 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:46:00.582050 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:00.572498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34777 (local address 127.14.237.2:43593)
0810 04:46:00.572682 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:00.572719 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:00.573331 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:00.573596 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:00.573600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:00.574319 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:00.574574 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:00.574582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.575273 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.575274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:00.576063 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:00.576065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.576439 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.576440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:00.576572 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:00.577094 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:00.577102 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:00.578789 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:00.580468 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:00.580475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:00.580477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:00.580703 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:00.580915 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:00.580916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:00.580916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:00.581030 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:00.581335 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:00.581342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:00.581490 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 04:46:00.581595 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 04:46:00.581784 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260810 04:46:00.582545 19901 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34777
I20260810 04:46:00.582649 19901 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
I20260810 04:46:00.582862 19901 heartbeater.cc:511] Master 127.14.237.62:34777 requested a full tablet report, sending...
I20260810 04:46:00.583344 19375 ts_manager.cc:194] Registered new tserver with Master: bf6bd32a6ee246e793150c9d1b9c9733 (127.14.237.2:43095)
I20260810 04:46:00.583671 19375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:43593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:46:00.599740 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34777
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:34933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:00.680465 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:00.680660 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:00.680682 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:00.681895 19913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:00.681955 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:00.682008 19913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:00.683468 19913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:34933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:00.683713 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:00.683949 19913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:00.688944 19922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:00.688938 19920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:00.688938 19919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:00.689395 19913 server_base.cc:1034] running on GCE node
I20260810 04:46:00.689603 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:00.689821 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:00.690982 19913 hybrid_clock.cc:648] HybridClock initialized: now 1786337160690972 us; error 28 us; skew 500 ppm
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:00.693797 19913 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:46:00.694303 19913 webserver.cc:460] Webserver started at http://127.14.237.3:45975/ using document root <none> and password file <none>
I20260810 04:46:00.694559 19913 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:00.694638 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:00.694785 19913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:00.695820 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "582a7c527371484a9f68aac87e03bef1"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "b57d84fe409bd9d3eed596c8ff6b32d2"
server_key_iv: "92854a411a91fad7141017c2787c15dc"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.696193 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "582a7c527371484a9f68aac87e03bef1"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "b57d84fe409bd9d3eed596c8ff6b32d2"
server_key_iv: "92854a411a91fad7141017c2787c15dc"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.697582 19913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:00.698274 19934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.698444 19913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:00.698549 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "582a7c527371484a9f68aac87e03bef1"
format_stamp: "Formatted at 2026-08-10 04:46:00 on dist-test-slave-8hhm"
server_key: "b57d84fe409bd9d3eed596c8ff6b32d2"
server_key_iv: "92854a411a91fad7141017c2787c15dc"
server_key_version: "encryptionkey@0"
I20260810 04:46:00.698621 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:00.712193 19913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:00.713985 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:00.714140 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:00.714399 19913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:00.714717 19913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:00.714771 19913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.714809 19913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:00.714835 19913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:00.720353 19913 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:39249
I20260810 04:46:00.720427 20132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:39249 every 8 connection(s)
I20260810 04:46:00.720772 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:00.724522 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19913
I20260810 04:46:00.724722 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:00.724892 15284 external_mini_cluster.cc:1428] Setting key 9f57aed46ab1f3f9c4ffbce2d54118f8
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337160, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:00.729417 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:00.721527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:54381 (local address 127.14.237.62:34777)
0810 04:46:00.721570 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:00.721572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:00.721793 (+ 221us) server_negotiation.cc:407] Connection header received
0810 04:46:00.722239 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:00.722240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:00.722251 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:00.722308 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:00.723194 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.723662 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.724596 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.724850 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.724935 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:00.727193 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:00.727194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:00.727194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:00.727201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:00.728280 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:00.728622 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:00.728622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:00.728623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:00.728673 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:00.728879 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:00.728879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:00.728879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:00.728971 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:00.729018 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:00.729281 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 04:46:00.729357 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:46:00.729627 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:00.721481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34777 (local address 127.14.237.3:54381)
0810 04:46:00.721640 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:00.721682 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:00.722156 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:00.722359 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:00.722362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:00.722923 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:00.723119 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:00.723123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.723745 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.723746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:00.724487 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:00.724488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:00.724947 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:00.724948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:00.725108 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:00.725702 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:00.725709 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:00.727098 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:00.728344 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:00.728349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:00.728351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:00.728557 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:00.728730 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:00.728731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:00.728731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:00.728825 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:00.729021 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:00.729025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:00.729127 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 04:46:00.729204 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:46:00.729389 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260810 04:46:00.730098 20135 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34777
I20260810 04:46:00.730175 20135 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:00.730374 20135 heartbeater.cc:511] Master 127.14.237.62:34777 requested a full tablet report, sending...
I20260810 04:46:00.730729 19375 ts_manager.cc:194] Registered new tserver with Master: 582a7c527371484a9f68aac87e03bef1 (127.14.237.3:39249)
I20260810 04:46:00.731048 19375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:54381
I20260810 04:46:00.736724 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:46:00.749570 19454 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:00.748637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.14.237.62:34777)
0810 04:46:00.748710 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:00.748712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:00.748767 (+ 55us) server_negotiation.cc:407] Connection header received
0810 04:46:00.748823 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:00.748824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:00.748835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:00.748888 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:00.749449 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47450: BlockingRecv error: recv got EOF from 127.0.0.1:47450 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:46:00.750397 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19475
I20260810 04:46:00.769503 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19687
I20260810 04:46:00.780440 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19913
I20260810 04:46:00.792939 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19312
2026-08-10T04:46:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2132 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[20158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[20158](info): set up 2 sockets
Aug 10 04:46:00 dist-test-slave-8hhm krb5kdc[20158](info): commencing operation
krb5kdc: starting...
W20260810 04:46:02.094606 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:02 dist-test-slave-8hhm krb5kdc[20158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:02.218281 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:39901
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:39901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:02.303970 20179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:02.304240 20179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:02.304266 20179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:02.305389 20179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:02.305430 20179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:02.305446 20179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:02.305460 20179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:02.305474 20179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:02.306950 20179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:39901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:39901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:02.307161 20179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:02.307356 20179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:02.312435 20190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:02.312454 20195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:02.312582 20179 server_base.cc:1034] running on GCE node
W20260810 04:46:02.312455 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:02.312901 20179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:02.313289 20179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:02.314474 20179 hybrid_clock.cc:648] HybridClock initialized: now 1786337162314447 us; error 40 us; skew 500 ppm
Aug 10 04:46:02 dist-test-slave-8hhm krb5kdc[20158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337162, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:02.319867 20179 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:02.320395 20179 webserver.cc:460] Webserver started at http://127.14.237.62:42413/ using document root <none> and password file <none>
I20260810 04:46:02.320621 20179 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:02.320667 20179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:02.320839 20179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:02.321964 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "71d9423ff75a46159297020ee445fc4e"
format_stamp: "Formatted at 2026-08-10 04:46:02 on dist-test-slave-8hhm"
server_key: "c2b4a77384ec8f2225add3400bb4241e"
server_key_iv: "80ff88550b95fe032682add454ced28f"
server_key_version: "encryptionkey@0"
I20260810 04:46:02.322351 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "71d9423ff75a46159297020ee445fc4e"
format_stamp: "Formatted at 2026-08-10 04:46:02 on dist-test-slave-8hhm"
server_key: "c2b4a77384ec8f2225add3400bb4241e"
server_key_iv: "80ff88550b95fe032682add454ced28f"
server_key_version: "encryptionkey@0"
I20260810 04:46:02.323698 20179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:46:02.324440 20208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:02.324661 20179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:02.324774 20179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "71d9423ff75a46159297020ee445fc4e"
format_stamp: "Formatted at 2026-08-10 04:46:02 on dist-test-slave-8hhm"
server_key: "c2b4a77384ec8f2225add3400bb4241e"
server_key_iv: "80ff88550b95fe032682add454ced28f"
server_key_version: "encryptionkey@0"
I20260810 04:46:02.324848 20179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:02.365332 20179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:02.367395 20179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:02.367569 20179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:02.371747 20179 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:39901
I20260810 04:46:02.371845 20297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:39901 every 8 connection(s)
I20260810 04:46:02.372392 20179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:02.373082 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:02.374414 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20179
I20260810 04:46:02.374531 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:02.374724 15284 external_mini_cluster.cc:1428] Setting key e89e8d59aec6a5080f87f96a219e0e34
I20260810 04:46:02.375909 20298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e: Bootstrap starting.
I20260810 04:46:02.376695 20298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:02.377035 20298 log.cc:824] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:02.377739 20298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e: No bootstrap required, opened a new log
I20260810 04:46:02.380108 20298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d9423ff75a46159297020ee445fc4e" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39901 } }
I20260810 04:46:02.380301 20298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:02.380342 20298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d9423ff75a46159297020ee445fc4e, State: Initialized, Role: FOLLOWER
I20260810 04:46:02.380484 20298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d9423ff75a46159297020ee445fc4e" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39901 } }
I20260810 04:46:02.380577 20298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:02.380611 20298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:02.380664 20298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:02.381429 20298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d9423ff75a46159297020ee445fc4e" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39901 } }
I20260810 04:46:02.381521 20298 leader_election.cc:302] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71d9423ff75a46159297020ee445fc4e; no voters:
I20260810 04:46:02.381698 20298 leader_election.cc:288] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:02.381757 20306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 1 FOLLOWER]: Leader election won for term 1
Aug 10 04:46:02 dist-test-slave-8hhm krb5kdc[20158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:02.381917 20306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [term 1 LEADER]: Becoming Leader. State: Replica: 71d9423ff75a46159297020ee445fc4e, State: Running, Role: LEADER
I20260810 04:46:02.381989 20298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:02.382014 20306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d9423ff75a46159297020ee445fc4e" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39901 } }
I20260810 04:46:02.382408 20306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d9423ff75a46159297020ee445fc4e. Latest consensus state: current_term: 1 leader_uuid: "71d9423ff75a46159297020ee445fc4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d9423ff75a46159297020ee445fc4e" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39901 } } }
I20260810 04:46:02.382421 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71d9423ff75a46159297020ee445fc4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d9423ff75a46159297020ee445fc4e" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39901 } } }
I20260810 04:46:02.382666 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:02.382508 20306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:02.382956 20322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:02.383483 20322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:02.386219 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:02.375271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.14.237.62:39901)
0810 04:46:02.375457 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:02.375465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:02.375490 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:46:02.375893 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:02.375900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:02.375946 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:02.376100 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:02.376902 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:02.378048 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:02.379066 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:02.379453 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:02.379531 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:02.382489 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:02.382493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:02.382493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:02.382506 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:02.384520 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:02.384929 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:02.384932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:02.384933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:02.385035 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:02.385238 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:02.385239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:02.385240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:02.385531 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:02.385689 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:02.385851 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 04:46:02.386018 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260810 04:46:02.386634 20322 catalog_manager.cc:1347] Generated new cluster ID: 7ca56b61e82741fc9658fc582e5a0183
I20260810 04:46:02.386693 20322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:02.387102 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:46:02.410745 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:02.406526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.14.237.62:39901)
0810 04:46:02.406580 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:02.406583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:02.406691 (+ 108us) server_negotiation.cc:407] Connection header received
0810 04:46:02.406735 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:02.406736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:02.406749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:02.406828 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:02.407256 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:02.407992 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:02.408996 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:02.409398 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:02.409467 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:02.410233 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:02.410235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:02.410235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:02.410486 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:02.410506 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:02.410513 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:02.410576 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:02.410632 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 04:46:02.410658 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:46:02.423817 20322 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:02.424356 20322 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:02.430924 20322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71d9423ff75a46159297020ee445fc4e: Generated new TSK 0
I20260810 04:46:02.431118 20322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:46:02.433514 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20179
2026-08-10T04:46:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1639 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:02 dist-test-slave-8hhm krb5kdc[20356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:02 dist-test-slave-8hhm krb5kdc[20356](info): set up 2 sockets
Aug 10 04:46:02 dist-test-slave-8hhm krb5kdc[20356](info): commencing operation
krb5kdc: starting...
W20260810 04:46:03.694792 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:03 dist-test-slave-8hhm krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:03.815865 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:34825
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:34825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:03.899147 20373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:03.899358 20373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:03.899382 20373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:03.900524 20373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:03.900566 20373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:03.900580 20373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:03.900593 20373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:03.900604 20373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:03.902017 20373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:34825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:03.902253 20373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:03.902434 20373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:03.907177 20386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:03.907317 20373 server_base.cc:1034] running on GCE node
W20260810 04:46:03.907260 20383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:03.907182 20392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:03.907738 20373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:03.907984 20373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:03.909142 20373 hybrid_clock.cc:648] HybridClock initialized: now 1786337163909122 us; error 35 us; skew 500 ppm
Aug 10 04:46:03 dist-test-slave-8hhm krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337163, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:03.911695 20373 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:03.912112 20373 webserver.cc:460] Webserver started at http://127.14.237.62:46091/ using document root <none> and password file <none>
I20260810 04:46:03.912293 20373 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:03.912333 20373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:03.912497 20373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:03.913611 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "990520c3ebe242ef9e66a41e44cef4f1"
format_stamp: "Formatted at 2026-08-10 04:46:03 on dist-test-slave-8hhm"
server_key: "9138b5038d8066a7272af0b29d51ad43"
server_key_iv: "f3eb5b0c1a40f41b265137ee09337565"
server_key_version: "encryptionkey@0"
I20260810 04:46:03.913921 20373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "990520c3ebe242ef9e66a41e44cef4f1"
format_stamp: "Formatted at 2026-08-10 04:46:03 on dist-test-slave-8hhm"
server_key: "9138b5038d8066a7272af0b29d51ad43"
server_key_iv: "f3eb5b0c1a40f41b265137ee09337565"
server_key_version: "encryptionkey@0"
I20260810 04:46:03.915177 20373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:46:03.915855 20402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:03.916031 20373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:03.916128 20373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "990520c3ebe242ef9e66a41e44cef4f1"
format_stamp: "Formatted at 2026-08-10 04:46:03 on dist-test-slave-8hhm"
server_key: "9138b5038d8066a7272af0b29d51ad43"
server_key_iv: "f3eb5b0c1a40f41b265137ee09337565"
server_key_version: "encryptionkey@0"
I20260810 04:46:03.916208 20373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:03.937008 20373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:03.938978 20373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:03.939152 20373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:03.943246 20373 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:34825
I20260810 04:46:03.943292 20485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:34825 every 8 connection(s)
I20260810 04:46:03.943683 20373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:03.944765 20486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:03.947109 20486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1: Bootstrap starting.
I20260810 04:46:03.947710 20486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:03.948022 20486 log.cc:824] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:03.948741 20486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1: No bootstrap required, opened a new log
I20260810 04:46:03.950367 20486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990520c3ebe242ef9e66a41e44cef4f1" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34825 } }
I20260810 04:46:03.950507 20486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:03.950551 20486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990520c3ebe242ef9e66a41e44cef4f1, State: Initialized, Role: FOLLOWER
I20260810 04:46:03.950749 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20373
I20260810 04:46:03.950731 20486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990520c3ebe242ef9e66a41e44cef4f1" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34825 } }
I20260810 04:46:03.950814 20486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:03.950858 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:03.950863 20486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:03.950911 20486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:03.951061 15284 external_mini_cluster.cc:1428] Setting key bb129f29a7aa4c8d0d00da98b77b8769
I20260810 04:46:03.951895 20486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990520c3ebe242ef9e66a41e44cef4f1" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34825 } }
I20260810 04:46:03.951987 20486 leader_election.cc:302] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 990520c3ebe242ef9e66a41e44cef4f1; no voters:
I20260810 04:46:03.952194 20486 leader_election.cc:288] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:03.952214 20501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:03.952356 20501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 990520c3ebe242ef9e66a41e44cef4f1, State: Running, Role: LEADER
I20260810 04:46:03.952442 20501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990520c3ebe242ef9e66a41e44cef4f1" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34825 } }
I20260810 04:46:03.952490 20486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:03.952749 20501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "990520c3ebe242ef9e66a41e44cef4f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990520c3ebe242ef9e66a41e44cef4f1" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34825 } } }
I20260810 04:46:03.952828 20501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:03.952754 20502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990520c3ebe242ef9e66a41e44cef4f1. Latest consensus state: current_term: 1 leader_uuid: "990520c3ebe242ef9e66a41e44cef4f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990520c3ebe242ef9e66a41e44cef4f1" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34825 } } }
I20260810 04:46:03.953006 20502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:03.953049 20505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:03.953737 20505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:03.955721 20505 catalog_manager.cc:1347] Generated new cluster ID: 8541b07005d2431abb521bdfbc19536c
I20260810 04:46:03.955782 20505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:46:03 dist-test-slave-8hhm krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:03.962265 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:03.951562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.14.237.62:34825)
0810 04:46:03.951777 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:03.951786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:03.951829 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:46:03.951923 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:03.951928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:03.951990 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:03.952134 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:03.952977 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:03.954391 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:03.955535 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:03.956092 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:03.956212 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:03.958577 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:03.958581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:03.958581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:03.958593 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:03.960287 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:03.960713 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:03.960720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:03.960721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:03.960823 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:03.961110 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:03.961112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:03.961112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:03.961495 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:03.961672 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:03.961974 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 04:46:03.962111 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260810 04:46:03.963555 20505 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:03.964239 20505 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:03.972854 20505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 990520c3ebe242ef9e66a41e44cef4f1: Generated new TSK 0
I20260810 04:46:03.973066 20505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:46:04.009841 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34825
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:04.105845 20532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:04.106042 20532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:04.106065 20532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:04.107218 20532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:04.107267 20532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:04.107292 20532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:04.108878 20532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:04.109124 20532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:04.109409 20532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:04.114519 20542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:04.114533 20540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:04.114744 20532 server_base.cc:1034] running on GCE node
W20260810 04:46:04.114533 20544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:04.115002 20532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:04.115245 20532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:04.116379 20532 hybrid_clock.cc:648] HybridClock initialized: now 1786337164116373 us; error 32 us; skew 500 ppm
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:04.119124 20532 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:46:04.119519 20532 webserver.cc:460] Webserver started at http://127.14.237.1:36073/ using document root <none> and password file <none>
I20260810 04:46:04.119730 20532 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:04.119772 20532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:04.119956 20532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:04.120961 20532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "8c7ece899dc14eca9b30ce315af7fdcd"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "dbdaa932b597c1b8195c9e449627d44e"
server_key_iv: "bdc7b279e3d6aee90872ca849e4ee5eb"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.121347 20532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "8c7ece899dc14eca9b30ce315af7fdcd"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "dbdaa932b597c1b8195c9e449627d44e"
server_key_iv: "bdc7b279e3d6aee90872ca849e4ee5eb"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.122676 20532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:04.123344 20558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.123528 20532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.123631 20532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "8c7ece899dc14eca9b30ce315af7fdcd"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "dbdaa932b597c1b8195c9e449627d44e"
server_key_iv: "bdc7b279e3d6aee90872ca849e4ee5eb"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.123703 20532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:04.138199 20532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:04.140646 20532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:04.140822 20532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:04.141067 20532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:04.141435 20532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:04.141492 20532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.141530 20532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:04.141556 20532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.147332 20532 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:44739
I20260810 04:46:04.147377 20752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:44739 every 8 connection(s)
I20260810 04:46:04.147744 20532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:04.156467 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20532
I20260810 04:46:04.156611 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:04.156840 15284 external_mini_cluster.cc:1428] Setting key f1f083189fbdeb923376b46ebc0dfe64
I20260810 04:46:04.157393 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:04.148518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:37515 (local address 127.14.237.62:34825)
0810 04:46:04.148582 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:04.148584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:04.148816 (+ 232us) server_negotiation.cc:407] Connection header received
0810 04:46:04.149405 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:04.149406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:04.149420 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:04.149479 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:04.150412 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.150911 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.151947 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.152159 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.152211 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:04.154529 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:04.154530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:04.154531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:04.154556 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:04.155886 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:04.156283 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:04.156285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:04.156285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:04.156357 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:04.156609 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:04.156611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:04.156611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:04.156774 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:04.156847 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:04.157193 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 04:46:04.157322 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:46:04.157624 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:04.148487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34825 (local address 127.14.237.1:37515)
0810 04:46:04.148679 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:04.148723 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:04.149311 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:04.149536 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:04.149539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:04.150134 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:04.150329 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:04.150335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.150995 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.150995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:04.151822 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:04.151824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.152234 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.152235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:04.152339 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:04.152823 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:04.152832 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:04.154440 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:04.155967 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:04.155973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:04.155975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:04.156200 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:04.156430 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:04.156431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:04.156431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:04.156534 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:04.156850 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:04.156858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:04.157021 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 04:46:04.157137 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 04:46:04.157360 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260810 04:46:04.158315 20753 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34825
I20260810 04:46:04.158440 20753 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:04.158639 20753 heartbeater.cc:511] Master 127.14.237.62:34825 requested a full tablet report, sending...
I20260810 04:46:04.159214 20423 ts_manager.cc:194] Registered new tserver with Master: 8c7ece899dc14eca9b30ce315af7fdcd (127.14.237.1:44739)
I20260810 04:46:04.159863 20423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:37515
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:46:04.184412 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34825
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:04.270803 20762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:04.271098 20762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:04.271137 20762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:04.273674 20762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:04.273747 20762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:04.273802 20762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:04.275484 20762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:04.276083 20762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:04.276284 20762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:04.282698 20772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:04.282713 20771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:04.283048 20774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:04.285168 20762 server_base.cc:1034] running on GCE node
I20260810 04:46:04.285430 20762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:04.285748 20762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:04.286909 20762 hybrid_clock.cc:648] HybridClock initialized: now 1786337164286894 us; error 34 us; skew 500 ppm
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:04.289994 20762 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:46:04.290598 20762 webserver.cc:460] Webserver started at http://127.14.237.2:35203/ using document root <none> and password file <none>
I20260810 04:46:04.290855 20762 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:04.290926 20762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:04.291076 20762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:04.292600 20762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "5eb657afbe3e4b7f9c0935686dcf476e"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "6fec814466bcfe921e2ec06457c7e796"
server_key_iv: "d403bcd0b492fafac36ea7f487bba9d7"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.293072 20762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "5eb657afbe3e4b7f9c0935686dcf476e"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "6fec814466bcfe921e2ec06457c7e796"
server_key_iv: "d403bcd0b492fafac36ea7f487bba9d7"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.295141 20762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:04.296186 20784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.296391 20762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:04.296491 20762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "5eb657afbe3e4b7f9c0935686dcf476e"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "6fec814466bcfe921e2ec06457c7e796"
server_key_iv: "d403bcd0b492fafac36ea7f487bba9d7"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.296568 20762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:04.324765 20762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:04.327423 20762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:04.327606 20762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:04.327857 20762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:04.328171 20762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:04.328228 20762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.328289 20762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:04.328320 20762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.334285 20762 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:45503
I20260810 04:46:04.334327 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:45503 every 8 connection(s)
I20260810 04:46:04.334676 20762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:04.341041 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20762
I20260810 04:46:04.341140 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:04.341358 15284 external_mini_cluster.cc:1428] Setting key 45c6ab6e4c96d4b83404ea4e7dedcdbc
I20260810 04:46:04.344834 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:04.335434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:34335 (local address 127.14.237.62:34825)
0810 04:46:04.335496 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:04.335498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:04.335707 (+ 209us) server_negotiation.cc:407] Connection header received
0810 04:46:04.336255 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:04.336257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:04.336269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:04.336327 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:04.337334 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.337833 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.338776 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.339050 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.339135 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:04.341658 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:04.341659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:04.341660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:04.341667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:04.343403 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:04.343818 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:04.343819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:04.343819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:04.343885 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:04.344131 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:04.344132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:04.344132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:04.344285 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:04.344360 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:04.344690 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 04:46:04.344769 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:46:04.345117 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:04.335405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34825 (local address 127.14.237.2:34335)
0810 04:46:04.335585 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:04.335621 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:04.336146 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:04.336400 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:04.336404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:04.336983 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:04.337179 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:04.337184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.337915 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.337916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:04.338701 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:04.338703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.339150 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.339151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:04.339302 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:04.339936 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:04.339944 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:04.341519 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:04.343507 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:04.343513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:04.343515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:04.343750 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:04.343951 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:04.343952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:04.343953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:04.344055 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:04.344358 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:04.344365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:04.344538 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 04:46:04.344670 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 04:46:04.344852 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260810 04:46:04.345743 20981 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34825
I20260810 04:46:04.345853 20981 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:04.346031 20981 heartbeater.cc:511] Master 127.14.237.62:34825 requested a full tablet report, sending...
I20260810 04:46:04.346403 20423 ts_manager.cc:194] Registered new tserver with Master: 5eb657afbe3e4b7f9c0935686dcf476e (127.14.237.2:45503)
I20260810 04:46:04.346776 20423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:34335
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:46:04.368793 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34825
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:04.456774 20996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:04.456979 20996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:04.457001 20996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:04.458209 20996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:04.458250 20996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:04.458276 20996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:04.459605 20996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:04.459797 20996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:04.459966 20996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:46:04.465004 20996 server_base.cc:1034] running on GCE node
W20260810 04:46:04.465041 21011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:04.465044 21005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:04.465170 21006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:04.465406 20996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:04.465633 20996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:04.466775 20996 hybrid_clock.cc:648] HybridClock initialized: now 1786337164466762 us; error 28 us; skew 500 ppm
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:04.469368 20996 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:46:04.469789 20996 webserver.cc:460] Webserver started at http://127.14.237.3:45651/ using document root <none> and password file <none>
I20260810 04:46:04.470003 20996 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:04.470046 20996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:04.470252 20996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:04.471266 20996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "674e4d7f0294466eb03a7afe61ea25a5"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "7e8b078ce7af2039a3cd210b84f8c322"
server_key_iv: "07fa5e2afb3e4e30d318109c3eaa39a4"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.471662 20996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "674e4d7f0294466eb03a7afe61ea25a5"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "7e8b078ce7af2039a3cd210b84f8c322"
server_key_iv: "07fa5e2afb3e4e30d318109c3eaa39a4"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.473145 20996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:04.473942 21021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.474140 20996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:04.474251 20996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "674e4d7f0294466eb03a7afe61ea25a5"
format_stamp: "Formatted at 2026-08-10 04:46:04 on dist-test-slave-8hhm"
server_key: "7e8b078ce7af2039a3cd210b84f8c322"
server_key_iv: "07fa5e2afb3e4e30d318109c3eaa39a4"
server_key_version: "encryptionkey@0"
I20260810 04:46:04.474329 20996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:04.482478 20996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:04.484352 20996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:04.484516 20996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:04.484757 20996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:04.485050 20996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:04.485101 20996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.485140 20996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:04.485164 20996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:04.491063 20996 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:40769
I20260810 04:46:04.491109 21226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:40769 every 8 connection(s)
I20260810 04:46:04.491420 20996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:04.494468 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20996
I20260810 04:46:04.494542 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:04.494711 15284 external_mini_cluster.cc:1428] Setting key 54a12da6cd850a1389e70b21aed2e908
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337164, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:04.500814 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:04.492155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:44265 (local address 127.14.237.62:34825)
0810 04:46:04.492203 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:04.492205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:04.492413 (+ 208us) server_negotiation.cc:407] Connection header received
0810 04:46:04.492933 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:04.492934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:04.492946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:04.493007 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:04.494007 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.494542 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.495851 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.496059 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.496115 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:04.498349 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:04.498350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:04.498351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:04.498357 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:04.499571 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:04.499923 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:04.499923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:04.499924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:04.499977 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:04.500202 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:04.500203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:04.500203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:04.500308 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:04.500356 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:04.500621 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 04:46:04.500679 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:46:04.501070 21231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:04.492129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34825 (local address 127.14.237.3:44265)
0810 04:46:04.492289 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:04.492326 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:04.492809 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:04.493067 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:04.493070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:04.493731 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:04.493923 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:04.493928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.494658 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.494659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:04.495783 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:04.495785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.496160 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.496161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:04.496259 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:04.496806 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:04.496820 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:04.498259 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:04.499640 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:04.499647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:04.499649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:04.499860 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:04.500030 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:04.500031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:04.500031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:04.500152 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:04.500358 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:04.500362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:04.500513 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 04:46:04.500596 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:46:04.500790 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260810 04:46:04.501643 21228 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34825
I20260810 04:46:04.501726 21228 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:04.501873 21228 heartbeater.cc:511] Master 127.14.237.62:34825 requested a full tablet report, sending...
I20260810 04:46:04.502172 20423 ts_manager.cc:194] Registered new tserver with Master: 674e4d7f0294466eb03a7afe61ea25a5 (127.14.237.3:40769)
I20260810 04:46:04.502573 20423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:44265
I20260810 04:46:04.506306 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[20356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337164, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[20356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337164, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:04.530421 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:04.523200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.14.237.62:34825)
0810 04:46:04.523302 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:04.523305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:04.523401 (+ 96us) server_negotiation.cc:407] Connection header received
0810 04:46:04.523511 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:04.523512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:04.523523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:04.523590 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:04.524176 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.524662 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.525446 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:04.525659 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:04.525737 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:04.528091 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:04.528092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:04.528092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:04.528100 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:04.529407 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:04.529720 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:04.529721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:04.529721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:04.529776 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:04.529935 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:04.529936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:04.529936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:04.530078 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:04.530155 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:04.530285 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 04:46:04.530351 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 04:46:04.530682 20423 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57328
I20260810 04:46:04.531114 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20532
I20260810 04:46:04.548403 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20762
I20260810 04:46:04.558354 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20996
I20260810 04:46:04.575033 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20373
2026-08-10T04:46:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2140 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[21244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[21244](info): set up 2 sockets
Aug 10 04:46:04 dist-test-slave-8hhm krb5kdc[21244](info): commencing operation
krb5kdc: starting...
W20260810 04:46:05.831629 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:05 dist-test-slave-8hhm krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:05.953922 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37745
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:37745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:06.036317 21267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:06.036511 21267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:06.036531 21267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:06.037719 21267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:06.037760 21267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:06.037774 21267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:06.037787 21267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:06.037843 21267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:06.039332 21267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:37745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:06.039592 21267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:06.039832 21267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:06.044744 21279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:06.044776 21281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:06.044750 21277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:06.044809 21267 server_base.cc:1034] running on GCE node
I20260810 04:46:06.045182 21267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:06.045516 21267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:06.046679 21267 hybrid_clock.cc:648] HybridClock initialized: now 1786337166046671 us; error 35 us; skew 500 ppm
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:06.049103 21267 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:06.049567 21267 webserver.cc:460] Webserver started at http://127.14.237.62:34509/ using document root <none> and password file <none>
I20260810 04:46:06.049815 21267 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:06.049881 21267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:06.050019 21267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:06.051039 21267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "08bd11441afd442e9d77b2a9dd737d41"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "7dff769e61429e62481e8c1d94b447ab"
server_key_iv: "e1ee31a33ef2f4b097da45dc770196c6"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.051396 21267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "08bd11441afd442e9d77b2a9dd737d41"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "7dff769e61429e62481e8c1d94b447ab"
server_key_iv: "e1ee31a33ef2f4b097da45dc770196c6"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.052747 21267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:06.053574 21293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.053779 21267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:06.053881 21267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "08bd11441afd442e9d77b2a9dd737d41"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "7dff769e61429e62481e8c1d94b447ab"
server_key_iv: "e1ee31a33ef2f4b097da45dc770196c6"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.053952 21267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:06.075320 21267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:06.077546 21267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:06.077713 21267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:06.081729 21267 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:37745
I20260810 04:46:06.081768 21381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:37745 every 8 connection(s)
I20260810 04:46:06.082309 21267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:06.083073 21382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:06.085474 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41: Bootstrap starting.
I20260810 04:46:06.086076 21382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:06.086405 21382 log.cc:824] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:06.087100 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41: No bootstrap required, opened a new log
I20260810 04:46:06.088686 21382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bd11441afd442e9d77b2a9dd737d41" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37745 } }
I20260810 04:46:06.088829 21382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:06.088883 21382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bd11441afd442e9d77b2a9dd737d41, State: Initialized, Role: FOLLOWER
I20260810 04:46:06.089025 21382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bd11441afd442e9d77b2a9dd737d41" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37745 } }
I20260810 04:46:06.089110 21382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:06.089135 21382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:06.089238 21382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:06.089269 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 21267
I20260810 04:46:06.089386 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:06.089581 15284 external_mini_cluster.cc:1428] Setting key 57d55cb44b68b4486234a637be9e6d81
I20260810 04:46:06.090492 21382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bd11441afd442e9d77b2a9dd737d41" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37745 } }
I20260810 04:46:06.090622 21382 leader_election.cc:302] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08bd11441afd442e9d77b2a9dd737d41; no voters:
I20260810 04:46:06.090788 21382 leader_election.cc:288] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:06.090839 21396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:06.091012 21396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [term 1 LEADER]: Becoming Leader. State: Replica: 08bd11441afd442e9d77b2a9dd737d41, State: Running, Role: LEADER
I20260810 04:46:06.091090 21382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:06.091113 21396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bd11441afd442e9d77b2a9dd737d41" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37745 } }
I20260810 04:46:06.091499 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08bd11441afd442e9d77b2a9dd737d41. Latest consensus state: current_term: 1 leader_uuid: "08bd11441afd442e9d77b2a9dd737d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bd11441afd442e9d77b2a9dd737d41" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37745 } } }
I20260810 04:46:06.091593 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:06.091540 21400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08bd11441afd442e9d77b2a9dd737d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08bd11441afd442e9d77b2a9dd737d41" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37745 } } }
I20260810 04:46:06.091910 21400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:06.091815 21404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:06.092516 21404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:06.094206 21404 catalog_manager.cc:1347] Generated new cluster ID: cb5e8be096a946ee958bb8f8b565f73b
I20260810 04:46:06.094300 21404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:06.102545 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.090100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.14.237.62:37745)
0810 04:46:06.090312 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:06.090322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:06.090358 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:46:06.090478 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:06.090487 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:06.090557 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:06.090702 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:06.091586 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.093182 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.094008 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.094698 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.094827 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.098138 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:06.098143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:06.098143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:06.098158 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:06.100308 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.100763 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.100767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.100768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.100897 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.101152 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.101153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.101153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.101541 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:06.101742 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:06.102133 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 04:46:06.102299 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":147,"thread_start_us":117,"threads_started":1}
I20260810 04:46:06.121066 21404 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:06.121974 21404 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:06.131749 21404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08bd11441afd442e9d77b2a9dd737d41: Generated new TSK 0
I20260810 04:46:06.131975 21404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:46:06.172477 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37745
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:06.275629 21436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:06.275838 21436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:06.275861 21436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:06.276978 21436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:06.277015 21436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:06.277040 21436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:06.278626 21436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:06.278825 21436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:06.279004 21436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:06.284190 21444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:06.284176 21449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:06.284217 21436 server_base.cc:1034] running on GCE node
W20260810 04:46:06.284176 21445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:06.284564 21436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:06.284812 21436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:06.285950 21436 hybrid_clock.cc:648] HybridClock initialized: now 1786337166285939 us; error 27 us; skew 500 ppm
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:06.288599 21436 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:46:06.289023 21436 webserver.cc:460] Webserver started at http://127.14.237.1:35081/ using document root <none> and password file <none>
I20260810 04:46:06.289330 21436 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:06.289377 21436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:06.289520 21436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:06.290565 21436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "593d319b7ae14995bc9b2bc748fb1b82"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "c5735224e42af888832baf82cf86113b"
server_key_iv: "ab895c14ad9e0b44c6b8d9d32eb6515b"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.290930 21436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "593d319b7ae14995bc9b2bc748fb1b82"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "c5735224e42af888832baf82cf86113b"
server_key_iv: "ab895c14ad9e0b44c6b8d9d32eb6515b"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.292469 21436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:06.293207 21457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.293416 21436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:06.293507 21436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "593d319b7ae14995bc9b2bc748fb1b82"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "c5735224e42af888832baf82cf86113b"
server_key_iv: "ab895c14ad9e0b44c6b8d9d32eb6515b"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.293581 21436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:06.304776 21436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:06.306852 21436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:06.306967 21436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:06.307152 21436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:06.307433 21436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:06.307471 21436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.307493 21436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:06.307507 21436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.313251 21436 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:45113
I20260810 04:46:06.313314 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:45113 every 8 connection(s)
I20260810 04:46:06.313648 21436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:06.318521 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 21436
I20260810 04:46:06.318608 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:06.318823 15284 external_mini_cluster.cc:1428] Setting key ef59780ece00d2a2a90185a8e5ac3b11
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:06.324123 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.314476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:39819 (local address 127.14.237.62:37745)
0810 04:46:06.314584 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:06.314586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:06.314758 (+ 172us) server_negotiation.cc:407] Connection header received
0810 04:46:06.315732 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:06.315734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:06.315747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:06.315799 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:06.316441 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.316976 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.318046 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.318268 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.318355 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.320684 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:06.320686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:06.320686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:06.320695 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:06.322454 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.323076 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.323077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.323077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.323160 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.323475 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.323475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.323476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.323574 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:06.323626 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:06.323973 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 04:46:06.324040 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 04:46:06.324499 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.314387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37745 (local address 127.14.237.1:39819)
0810 04:46:06.314630 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:06.314668 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:06.315604 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:06.315875 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:06.315878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:06.316138 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:06.316323 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:06.316328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.317088 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.317089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:06.317948 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:06.317951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.318386 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.318387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:06.318507 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.319100 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:06.319109 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:06.320588 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:06.322554 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:06.322567 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:06.322569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:06.322989 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:06.323233 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:06.323234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:06.323234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:06.323401 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:06.323639 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:06.323647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:06.323831 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 04:46:06.323958 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 04:46:06.324191 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260810 04:46:06.325102 21656 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37745
I20260810 04:46:06.325241 21656 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:06.325428 21656 heartbeater.cc:511] Master 127.14.237.62:37745 requested a full tablet report, sending...
I20260810 04:46:06.326013 21323 ts_manager.cc:194] Registered new tserver with Master: 593d319b7ae14995bc9b2bc748fb1b82 (127.14.237.1:45113)
I20260810 04:46:06.326633 21323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:39819
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:46:06.348165 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37745
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:06.432283 21662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:06.432489 21662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:06.432511 21662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:06.433691 21662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:06.433741 21662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:06.433768 21662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:06.435209 21662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:06.435390 21662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:06.435563 21662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:06.440655 21668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:06.440672 21662 server_base.cc:1034] running on GCE node
W20260810 04:46:06.440637 21667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:06.440630 21670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:06.441005 21662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:06.441334 21662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:06.442498 21662 hybrid_clock.cc:648] HybridClock initialized: now 1786337166442477 us; error 50 us; skew 500 ppm
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:06.445250 21662 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:46:06.445694 21662 webserver.cc:460] Webserver started at http://127.14.237.2:39527/ using document root <none> and password file <none>
I20260810 04:46:06.445917 21662 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:06.445984 21662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:06.446137 21662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:06.447180 21662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "43dd4740656c4d2eadca3125d53124de"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "d9c5d2a00403123cafd1e3a7080aee13"
server_key_iv: "820c73a265a871135187ad28016bc7c4"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.447552 21662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "43dd4740656c4d2eadca3125d53124de"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "d9c5d2a00403123cafd1e3a7080aee13"
server_key_iv: "820c73a265a871135187ad28016bc7c4"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.448951 21662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:06.449767 21687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.449976 21662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:46:06.450115 21662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "43dd4740656c4d2eadca3125d53124de"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "d9c5d2a00403123cafd1e3a7080aee13"
server_key_iv: "820c73a265a871135187ad28016bc7c4"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.450202 21662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:06.497084 21662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:06.499148 21662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:06.499339 21662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:06.499598 21662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:06.499931 21662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:06.499985 21662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.500028 21662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:06.500061 21662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.505892 21662 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:45329
I20260810 04:46:06.505940 21879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:45329 every 8 connection(s)
I20260810 04:46:06.506264 21662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:06.514055 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 21662
I20260810 04:46:06.514174 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:06.514391 15284 external_mini_cluster.cc:1428] Setting key f3eff88a2e29381685fbc98d2220c439
I20260810 04:46:06.516126 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.506955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37745 (local address 127.14.237.2:34079)
0810 04:46:06.507134 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:06.507178 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:06.508106 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:06.508337 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:06.508340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:06.508603 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:06.508805 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:06.508809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.509537 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.509538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:06.510341 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:06.510343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.510672 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.510673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:06.510784 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.511313 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:06.511321 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:06.512772 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:06.514606 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:06.514613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:06.514615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:06.514899 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:06.515148 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:06.515149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:06.515149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:06.515310 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:06.515646 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:06.515651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:06.515765 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 04:46:06.515847 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:46:06.515964 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260810 04:46:06.516125 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.506976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:34079 (local address 127.14.237.62:37745)
0810 04:46:06.507040 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:06.507043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:06.507266 (+ 223us) server_negotiation.cc:407] Connection header received
0810 04:46:06.508216 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:06.508217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:06.508228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:06.508276 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:06.508879 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.509458 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.510422 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.510605 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.510654 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.512861 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:06.512863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:06.512863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:06.512870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:06.514524 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.514970 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.514971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.514971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.515039 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.515382 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.515383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.515383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.515558 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:06.515654 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:06.515895 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 04:46:06.516001 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:46:06.516649 21880 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37745
I20260810 04:46:06.516765 21880 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:06.516976 21880 heartbeater.cc:511] Master 127.14.237.62:37745 requested a full tablet report, sending...
I20260810 04:46:06.517467 21323 ts_manager.cc:194] Registered new tserver with Master: 43dd4740656c4d2eadca3125d53124de (127.14.237.2:45329)
I20260810 04:46:06.517982 21323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:34079
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:46:06.540489 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37745
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:06.624464 21886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:06.624698 21886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:06.624749 21886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:06.625976 21886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:06.626040 21886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:06.626088 21886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:06.627571 21886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:06.627843 21886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:06.628058 21886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:06.633257 21895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:06.633375 21896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:06.633285 21901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:06.633520 21886 server_base.cc:1034] running on GCE node
I20260810 04:46:06.633718 21886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:06.633982 21886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:06.635146 21886 hybrid_clock.cc:648] HybridClock initialized: now 1786337166635138 us; error 31 us; skew 500 ppm
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:06.637871 21886 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:46:06.638243 21886 webserver.cc:460] Webserver started at http://127.14.237.3:45121/ using document root <none> and password file <none>
I20260810 04:46:06.638446 21886 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:06.638484 21886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:06.638569 21886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:06.639518 21886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "59bfdc1f3eba485fad15754041440ef4"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "565af8524fb36e2a0a8f0d0dfa362f21"
server_key_iv: "f2438706ed0b425c608c4903dad24033"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.639818 21886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "59bfdc1f3eba485fad15754041440ef4"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "565af8524fb36e2a0a8f0d0dfa362f21"
server_key_iv: "f2438706ed0b425c608c4903dad24033"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.641067 21886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:46:06.641811 21911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.641970 21886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.642051 21886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "59bfdc1f3eba485fad15754041440ef4"
format_stamp: "Formatted at 2026-08-10 04:46:06 on dist-test-slave-8hhm"
server_key: "565af8524fb36e2a0a8f0d0dfa362f21"
server_key_iv: "f2438706ed0b425c608c4903dad24033"
server_key_version: "encryptionkey@0"
I20260810 04:46:06.642120 21886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:06.667743 21886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:06.669797 21886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:06.669966 21886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:06.670224 21886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:06.670545 21886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:06.670578 21886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.670603 21886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:06.670653 21886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:06.676438 21886 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:39023
I20260810 04:46:06.676508 22104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:39023 every 8 connection(s)
I20260810 04:46:06.676863 21886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:06.685683 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.677633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:34991 (local address 127.14.237.62:37745)
0810 04:46:06.677695 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:06.677697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:06.677911 (+ 214us) server_negotiation.cc:407] Connection header received
0810 04:46:06.678853 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:06.678854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:06.678864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:06.678925 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:06.679501 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.680005 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.680938 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.681120 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.681174 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.683316 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:06.683318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:06.683318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:06.683325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:06.684527 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.684895 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.684896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.684896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.684946 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.685162 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.685162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.685163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.685281 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:06.685331 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:06.685550 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 04:46:06.685609 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:46:06.685855 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.677602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37745 (local address 127.14.237.3:34991)
0810 04:46:06.677785 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:06.677827 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:06.678765 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:06.678981 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:06.678984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:06.679241 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:06.679428 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:06.679432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.680083 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.680084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:06.680864 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:06.680866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.681192 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.681193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:06.681356 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.681873 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:06.681881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:06.683228 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:06.684599 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:06.684605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:06.684606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:06.684832 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:06.685008 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:06.685009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:06.685009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:06.685108 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:06.685324 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:06.685329 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:06.685441 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 04:46:06.685525 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 04:46:06.685644 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260810 04:46:06.686021 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 21886
I20260810 04:46:06.686116 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:06.686334 15284 external_mini_cluster.cc:1428] Setting key 7c70d2786599440020a52727d01c050b
I20260810 04:46:06.686381 22105 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37745
I20260810 04:46:06.686486 22105 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:06.686694 22105 heartbeater.cc:511] Master 127.14.237.62:37745 requested a full tablet report, sending...
I20260810 04:46:06.687119 21322 ts_manager.cc:194] Registered new tserver with Master: 59bfdc1f3eba485fad15754041440ef4 (127.14.237.3:39023)
I20260810 04:46:06.687520 21322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:34991
I20260810 04:46:06.697961 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:46:06.721120 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.714163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.14.237.1:45113)
0810 04:46:06.714340 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:06.714346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:06.714364 (+ 18us) server_negotiation.cc:407] Connection header received
0810 04:46:06.714391 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:06.714394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:06.714408 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:06.714474 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:06.714993 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.715692 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.716337 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.716618 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.716669 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.718636 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:06.718644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:06.718644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:06.718651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:06.720007 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.720356 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.720359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.720360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.720425 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.720574 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.720574 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.720574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.720710 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:06.720814 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:06.720867 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:46:06.720934 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[21244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:06.729403 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:06.722488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.14.237.62:37745)
0810 04:46:06.722569 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:06.722570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:06.722581 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:46:06.722609 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:06.722610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:06.722621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:06.722692 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:06.723130 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.723607 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.724234 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:06.724414 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:06.724473 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:06.726686 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:06.726688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:06.726688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:06.726696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:06.727994 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.728493 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.728494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.728495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.728561 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:06.728760 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:06.728761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:06.728761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:06.728944 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:06.729067 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:06.729178 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 04:46:06.729286 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 04:46:06.729686 21322 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53700
I20260810 04:46:06.730041 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 21436
I20260810 04:46:06.747916 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 21662
I20260810 04:46:06.759557 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 21886
I20260810 04:46:06.770923 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 21267
2026-08-10T04:46:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2198 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[22123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[22123](info): set up 2 sockets
Aug 10 04:46:06 dist-test-slave-8hhm krb5kdc[22123](info): commencing operation
krb5kdc: starting...
W20260810 04:46:08.026292 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:08.144210 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:40949
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:40949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:08.227164 22141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:08.227370 22141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:08.227391 22141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:08.228427 22141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:08.228461 22141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:08.228474 22141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:08.228487 22141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:08.228498 22141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:08.229918 22141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:40949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:40949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:08.230103 22141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:08.230278 22141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:46:08.235208 22141 server_base.cc:1034] running on GCE node
W20260810 04:46:08.235184 22154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:08.235169 22156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:08.235188 22153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:08.235653 22141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:08.235944 22141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:08.237124 22141 hybrid_clock.cc:648] HybridClock initialized: now 1786337168237112 us; error 32 us; skew 500 ppm
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:08.242738 22141 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:08.243103 22141 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:08.243155 22141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:08.243248 22141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:08.244287 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "29d6d773c8aa46f3899b903b3b041784"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "957a21248b0096de49b144af7f08eb5d"
server_key_iv: "27a3a63df1b9604660ef18a886562f66"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.244591 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "29d6d773c8aa46f3899b903b3b041784"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "957a21248b0096de49b144af7f08eb5d"
server_key_iv: "27a3a63df1b9604660ef18a886562f66"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.245929 22141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:08.246654 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.246850 22141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:46:08.246954 22141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "29d6d773c8aa46f3899b903b3b041784"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "957a21248b0096de49b144af7f08eb5d"
server_key_iv: "27a3a63df1b9604660ef18a886562f66"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.247025 22141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:08.266211 22141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:08.268612 22141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:08.268741 22141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:08.273005 22141 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:40949
I20260810 04:46:08.273072 22252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:40949 every 8 connection(s)
I20260810 04:46:08.273411 22141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:08.274483 22253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:08.276849 22253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784: Bootstrap starting.
I20260810 04:46:08.277489 22253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:08.277748 22253 log.cc:824] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:08.278378 22253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784: No bootstrap required, opened a new log
I20260810 04:46:08.279050 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 22141
I20260810 04:46:08.279176 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:08.279378 15284 external_mini_cluster.cc:1428] Setting key bf500b0ea12abcf4639b6e855522c177
I20260810 04:46:08.280544 22253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d6d773c8aa46f3899b903b3b041784" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40949 } }
I20260810 04:46:08.280735 22253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:08.280824 22253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d6d773c8aa46f3899b903b3b041784, State: Initialized, Role: FOLLOWER
I20260810 04:46:08.280965 22253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d6d773c8aa46f3899b903b3b041784" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40949 } }
I20260810 04:46:08.281060 22253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:08.281095 22253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:08.281154 22253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:08.281840 22253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d6d773c8aa46f3899b903b3b041784" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40949 } }
I20260810 04:46:08.281932 22253 leader_election.cc:302] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29d6d773c8aa46f3899b903b3b041784; no voters:
I20260810 04:46:08.282094 22253 leader_election.cc:288] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:08.282200 22263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:08.282385 22253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:08.282438 22263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [term 1 LEADER]: Becoming Leader. State: Replica: 29d6d773c8aa46f3899b903b3b041784, State: Running, Role: LEADER
I20260810 04:46:08.282603 22263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d6d773c8aa46f3899b903b3b041784" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40949 } }
I20260810 04:46:08.283197 22265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29d6d773c8aa46f3899b903b3b041784" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d6d773c8aa46f3899b903b3b041784" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40949 } } }
I20260810 04:46:08.283288 22265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:08.283442 22266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29d6d773c8aa46f3899b903b3b041784. Latest consensus state: current_term: 1 leader_uuid: "29d6d773c8aa46f3899b903b3b041784" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d6d773c8aa46f3899b903b3b041784" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40949 } } }
I20260810 04:46:08.283512 22266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:08.283522 22281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:08.284029 22281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:08.286789 22281 catalog_manager.cc:1347] Generated new cluster ID: 4751496284144e4d8b1b970790a12162
I20260810 04:46:08.286850 22281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:08.290568 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.279935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.14.237.62:40949)
0810 04:46:08.280111 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.280119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.280145 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:46:08.280518 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.280525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.280571 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.280688 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:08.281538 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.282813 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.283891 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.284616 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.284734 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.287458 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:08.287461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:08.287462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:08.287473 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:08.289021 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.289449 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.289452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.289452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.289556 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.289762 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.289763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.289763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.290005 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:08.290174 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.290247 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.290379 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260810 04:46:08.295074 22281 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:08.295819 22281 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:08.301702 22281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784: Generated new TSK 0
I20260810 04:46:08.301883 22281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:46:08.332676 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40949
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:08.414042 22298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:08.414275 22298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:08.414310 22298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:08.415413 22298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:08.415478 22298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:08.415517 22298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:08.416985 22298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:08.417265 22298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:08.417498 22298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:08.422432 22311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:08.422451 22315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:08.422451 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:08.422710 22298 server_base.cc:1034] running on GCE node
I20260810 04:46:08.422940 22298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:08.423208 22298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:08.424379 22298 hybrid_clock.cc:648] HybridClock initialized: now 1786337168424368 us; error 28 us; skew 500 ppm
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:08.426954 22298 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:46:08.427343 22298 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:08.427412 22298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:08.427543 22298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:08.428660 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "101f8543727048e2b0dadea4d5e8b088"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "040ed3cd6aa99c3ed2994292e460457c"
server_key_iv: "bdb031f1bd6951c6bddcd8e04b9737d2"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.429018 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "101f8543727048e2b0dadea4d5e8b088"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "040ed3cd6aa99c3ed2994292e460457c"
server_key_iv: "bdb031f1bd6951c6bddcd8e04b9737d2"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.430354 22298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:08.430998 22321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.431177 22298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:08.431269 22298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "101f8543727048e2b0dadea4d5e8b088"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "040ed3cd6aa99c3ed2994292e460457c"
server_key_iv: "bdb031f1bd6951c6bddcd8e04b9737d2"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.431349 22298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:08.444368 22298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:08.446277 22298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:08.446439 22298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:08.446722 22298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:08.447099 22298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:08.447149 22298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.447192 22298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:08.447217 22298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.453411 22298 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:35707
I20260810 04:46:08.453464 22505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:35707 every 8 connection(s)
I20260810 04:46:08.453809 22298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:08.458086 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 22298
I20260810 04:46:08.458169 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:08.458356 15284 external_mini_cluster.cc:1428] Setting key 2e24f9e74083b614f8b368b8ce4a6f56
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:08.463148 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.454513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40949 (local address 127.14.237.1:52839)
0810 04:46:08.454704 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.454739 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:08.455225 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:08.455433 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:08.455436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:08.455684 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:08.456188 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.456193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.456817 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.456818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.457611 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.457613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.457959 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.457960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.458052 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.458728 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:08.458737 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:08.460144 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:08.461824 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:08.461831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:08.461832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:08.462070 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:08.462285 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:08.462286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:08.462286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:08.462382 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:08.462683 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:08.462687 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:08.462795 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 04:46:08.462875 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:46:08.462988 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260810 04:46:08.463181 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.454504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:52839 (local address 127.14.237.62:40949)
0810 04:46:08.454586 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.454589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.454817 (+ 228us) server_negotiation.cc:407] Connection header received
0810 04:46:08.455313 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.455314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.455328 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.455381 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:08.456261 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.456743 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.457674 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.457865 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.457938 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.460237 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:08.460239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:08.460239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:08.460247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:08.461727 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.462143 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.462144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.462145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.462211 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.462441 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.462442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.462443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.462611 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:08.462685 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.462922 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.463033 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 04:46:08.463605 22508 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40949
I20260810 04:46:08.463711 22508 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:08.463888 22508 heartbeater.cc:511] Master 127.14.237.62:40949 requested a full tablet report, sending...
I20260810 04:46:08.464434 22189 ts_manager.cc:194] Registered new tserver with Master: 101f8543727048e2b0dadea4d5e8b088 (127.14.237.1:35707)
I20260810 04:46:08.465009 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:52839
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:46:08.485829 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40949
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:08.567713 22517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:08.567957 22517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:08.567998 22517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:08.569145 22517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:08.569209 22517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:08.569272 22517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:08.570782 22517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:08.571035 22517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:08.571261 22517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:46:08.576360 22517 server_base.cc:1034] running on GCE node
W20260810 04:46:08.576352 22528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:08.576344 22524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:08.576315 22525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:08.576812 22517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:08.577047 22517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:08.578187 22517 hybrid_clock.cc:648] HybridClock initialized: now 1786337168578172 us; error 32 us; skew 500 ppm
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:08.580682 22517 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:46:08.581027 22517 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:08.581076 22517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:08.581202 22517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:08.582340 22517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "8a4e25ba8dff4b6db58fff6e291c75a9"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "9fddda1740f0f5bdaa9b40b2c5880ab5"
server_key_iv: "b7cd7b15d751e851a2cd01a705bada07"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.582726 22517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "8a4e25ba8dff4b6db58fff6e291c75a9"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "9fddda1740f0f5bdaa9b40b2c5880ab5"
server_key_iv: "b7cd7b15d751e851a2cd01a705bada07"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.584137 22517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:08.584815 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.584980 22517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:08.585079 22517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "8a4e25ba8dff4b6db58fff6e291c75a9"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "9fddda1740f0f5bdaa9b40b2c5880ab5"
server_key_iv: "b7cd7b15d751e851a2cd01a705bada07"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.585157 22517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:08.605439 22517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:08.607285 22517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:08.607438 22517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:08.607657 22517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:08.607980 22517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:08.608032 22517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.608074 22517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:08.608100 22517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.614012 22517 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:33629
I20260810 04:46:08.614054 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:33629 every 8 connection(s)
I20260810 04:46:08.614315 22517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:08.620498 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 22517
I20260810 04:46:08.620604 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:08.620844 15284 external_mini_cluster.cc:1428] Setting key b5f7f03d6adadf9780b16a98efa2209f
I20260810 04:46:08.623456 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.614998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:57269 (local address 127.14.237.62:40949)
0810 04:46:08.615049 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.615052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.615272 (+ 220us) server_negotiation.cc:407] Connection header received
0810 04:46:08.615739 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.615740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.615750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.615799 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:08.616692 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.617181 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.618182 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.618390 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.618439 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.620508 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:08.620510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:08.620510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:08.620518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:08.622138 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.622531 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.622532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.622532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.622596 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.622831 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.622832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.622832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.622949 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:08.623017 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.623282 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.623366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:46:08.623657 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.614998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40949 (local address 127.14.237.2:57269)
0810 04:46:08.615154 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.615190 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:08.615656 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:08.615861 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:08.615864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:08.616122 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:08.616619 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.616624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.617321 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.617321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.618109 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.618111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.618455 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.618456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.618584 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.619066 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:08.619075 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:08.620404 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:08.622220 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:08.622227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:08.622229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:08.622446 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:08.622659 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:08.622660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:08.622660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:08.622763 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:08.623019 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:08.623025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:08.623156 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 04:46:08.623259 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:46:08.623412 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260810 04:46:08.624321 22735 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40949
I20260810 04:46:08.624400 22735 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:08.624558 22735 heartbeater.cc:511] Master 127.14.237.62:40949 requested a full tablet report, sending...
I20260810 04:46:08.624948 22189 ts_manager.cc:194] Registered new tserver with Master: 8a4e25ba8dff4b6db58fff6e291c75a9 (127.14.237.2:33629)
I20260810 04:46:08.625401 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:57269
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:46:08.648108 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40949
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:08.728426 22740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:08.728663 22740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:08.728701 22740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:08.729826 22740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:08.729888 22740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:08.729931 22740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:08.731400 22740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:08.731679 22740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:08.731910 22740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:08.736750 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:08.736820 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:08.736795 22746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:08.736927 22740 server_base.cc:1034] running on GCE node
I20260810 04:46:08.737257 22740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:08.737465 22740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:08.738627 22740 hybrid_clock.cc:648] HybridClock initialized: now 1786337168738618 us; error 38 us; skew 500 ppm
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:08.741003 22740 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:46:08.741425 22740 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:08.741493 22740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:08.741628 22740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:08.742635 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "0ad0661f516a4adf977654a1bca58a5c"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "fa5221efb55810a51be2a228d1b4101e"
server_key_iv: "cd3298c7d1072887b0e3a3d12b29a330"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.742991 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "0ad0661f516a4adf977654a1bca58a5c"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "fa5221efb55810a51be2a228d1b4101e"
server_key_iv: "cd3298c7d1072887b0e3a3d12b29a330"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.744243 22740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:08.744939 22761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.745103 22740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:08.745198 22740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "0ad0661f516a4adf977654a1bca58a5c"
format_stamp: "Formatted at 2026-08-10 04:46:08 on dist-test-slave-8hhm"
server_key: "fa5221efb55810a51be2a228d1b4101e"
server_key_iv: "cd3298c7d1072887b0e3a3d12b29a330"
server_key_version: "encryptionkey@0"
I20260810 04:46:08.745307 22740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:08.766106 22740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:08.767755 22740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:08.767901 22740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:08.768124 22740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:08.768425 22740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:08.768472 22740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.768505 22740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:08.768530 22740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:08.774359 22740 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:46113
I20260810 04:46:08.774405 22956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:46113 every 8 connection(s)
I20260810 04:46:08.774648 22740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:46:08 dist-test-slave-8hhm krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337168, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:08.783460 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 22740
I20260810 04:46:08.783612 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:08.783742 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.775300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:39933 (local address 127.14.237.62:40949)
0810 04:46:08.775343 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.775345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.775555 (+ 210us) server_negotiation.cc:407] Connection header received
0810 04:46:08.776039 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.776040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.776054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.776101 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:08.776970 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.777524 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.778814 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.779008 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.779080 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.781036 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:08.781037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:08.781038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:08.781045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:08.782320 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.782731 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.782732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.782732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.782792 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.783058 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.783059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.783059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.783179 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:08.783250 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.783546 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.783640 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:46:08.783829 15284 external_mini_cluster.cc:1428] Setting key d0780bc59f723a8f31c88802fb9e3a34
I20260810 04:46:08.783905 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.775293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40949 (local address 127.14.237.3:39933)
0810 04:46:08.775451 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.775484 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:08.775938 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:08.776163 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:08.776167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:08.776409 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:08.776874 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.776879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.777621 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.777622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.778692 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.778695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.779101 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.779102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.779236 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.779661 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:08.779669 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:08.780946 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:08.782396 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:08.782403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:08.782405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:08.782636 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:08.782861 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:08.782862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:08.782862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:08.782964 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:08.783253 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:08.783260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:08.783410 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 04:46:08.783508 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 04:46:08.783662 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260810 04:46:08.784399 22957 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40949
I20260810 04:46:08.784500 22957 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:08.784690 22957 heartbeater.cc:511] Master 127.14.237.62:40949 requested a full tablet report, sending...
I20260810 04:46:08.785087 22189 ts_manager.cc:194] Registered new tserver with Master: 0ad0661f516a4adf977654a1bca58a5c (127.14.237.3:46113)
I20260810 04:46:08.785504 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:39933
I20260810 04:46:08.795827 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:46:08.802837 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.797051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.14.237.62:40949)
0810 04:46:08.797130 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.797132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.797183 (+ 51us) server_negotiation.cc:407] Connection header received
0810 04:46:08.797291 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.797291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.797302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.797362 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:08.797862 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.798342 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.799001 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.799172 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.799220 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.800371 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:08.800372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:08.800372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:08.800378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:08.801774 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.802092 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.802094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.802094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.802176 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:08.802375 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:08.802376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:08.802376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:08.802513 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:08.802594 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.802691 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.802754 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 04:46:08.804261 22189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:46:08.805910 22189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:08.812335 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.808435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.14.237.2:33629)
0810 04:46:08.808892 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.808897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.808907 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:08.808945 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.808947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.808965 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.809176 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:08.809284 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.810262 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.811413 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.811921 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.812008 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.812071 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.812232 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.812269 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":82,"threads_started":1}
I20260810 04:46:08.812338 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.808246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:33629 (local address 127.0.0.1:40512)
0810 04:46:08.808544 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.808555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:08.808612 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:08.809066 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:08.809067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:08.809069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:08.809107 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.809109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.810366 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.810366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.811335 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.811337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.812009 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.812010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.812112 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.812129 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 04:46:08.812206 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:46:08.812267 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":53,"threads_started":1}
I20260810 04:46:08.813093 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.808433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:35707 (local address 127.0.0.1:39668)
0810 04:46:08.808612 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.808621 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:08.808659 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:08.809106 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:08.809107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:08.809110 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:08.809155 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.809157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.810014 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.810015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.811328 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.811331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.812856 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.812857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.812918 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.812922 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:08.812980 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:46:08.813022 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260810 04:46:08.813123 22613 tablet_service.cc:1467] Processing CreateTablet for tablet 211b6160fa734cb6b513d3138edfb893 (DEFAULT_TABLE table=test-table [id=cac9e5cab1f64b5caebbedccbdafc9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:08.813361 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.808667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.14.237.1:35707)
0810 04:46:08.808861 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.808865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.808875 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:08.808930 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.808932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.808948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.809019 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:08.809253 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.809912 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.812109 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.812768 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.812853 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.812910 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.812995 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.813029 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260810 04:46:08.813552 22613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211b6160fa734cb6b513d3138edfb893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:08.813979 22403 tablet_service.cc:1467] Processing CreateTablet for tablet 211b6160fa734cb6b513d3138edfb893 (DEFAULT_TABLE table=test-table [id=cac9e5cab1f64b5caebbedccbdafc9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:08.814260 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.808245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:46113 (local address 127.0.0.1:39324)
0810 04:46:08.808402 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.808446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:08.808511 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:08.808809 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:08.808813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:08.808815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:08.808898 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.808900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.810223 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.810224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.811331 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.811332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.814008 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.814009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.814071 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.814075 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:08.814140 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 04:46:08.814181 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":39,"thread_start_us":86,"threads_started":1}
I20260810 04:46:08.814325 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211b6160fa734cb6b513d3138edfb893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:08.814517 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.808329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.14.237.3:46113)
0810 04:46:08.808633 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.808638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.808648 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:08.808686 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.808688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.808703 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.808797 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:08.808994 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.810126 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.812884 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.813917 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.813993 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.814048 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.814231 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.814278 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":89,"threads_started":1}
I20260810 04:46:08.816568 22992 tablet_bootstrap.cc:492] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Bootstrap starting.
I20260810 04:46:08.816677 22843 tablet_service.cc:1467] Processing CreateTablet for tablet 211b6160fa734cb6b513d3138edfb893 (DEFAULT_TABLE table=test-table [id=cac9e5cab1f64b5caebbedccbdafc9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:08.817003 22843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211b6160fa734cb6b513d3138edfb893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:08.817173 22992 tablet_bootstrap.cc:654] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:08.817371 22993 tablet_bootstrap.cc:492] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088: Bootstrap starting.
I20260810 04:46:08.817938 22992 log.cc:824] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:08.818104 22993 tablet_bootstrap.cc:654] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:08.818717 22993 log.cc:824] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:08.818835 22992 tablet_bootstrap.cc:492] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: No bootstrap required, opened a new log
I20260810 04:46:08.818904 22992 ts_tablet_manager.cc:1397] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:46:08.819587 22993 tablet_bootstrap.cc:492] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088: No bootstrap required, opened a new log
I20260810 04:46:08.819654 22993 ts_tablet_manager.cc:1397] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:46:08.820197 22998 tablet_bootstrap.cc:492] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: Bootstrap starting.
I20260810 04:46:08.820776 22998 tablet_bootstrap.cc:654] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:08.821099 22992 raft_consensus.cc:348] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.821336 22992 raft_consensus.cc:374] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:08.821319 22998 log.cc:824] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:08.821385 22992 raft_consensus.cc:729] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4e25ba8dff4b6db58fff6e291c75a9, State: Initialized, Role: FOLLOWER
I20260810 04:46:08.821520 22992 consensus_queue.cc:260] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.821815 22992 ts_tablet_manager.cc:1428] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 04:46:08.821734 22993 raft_consensus.cc:348] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.821878 22993 raft_consensus.cc:374] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:08.821923 22993 raft_consensus.cc:729] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f8543727048e2b0dadea4d5e8b088, State: Initialized, Role: FOLLOWER
I20260810 04:46:08.821947 22735 heartbeater.cc:503] Master 127.14.237.62:40949 was elected leader, sending a full tablet report...
I20260810 04:46:08.822036 22993 consensus_queue.cc:260] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.822091 22998 tablet_bootstrap.cc:492] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: No bootstrap required, opened a new log
I20260810 04:46:08.822183 22998 ts_tablet_manager.cc:1397] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:46:08.822315 22993 ts_tablet_manager.cc:1428] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 04:46:08.822417 22508 heartbeater.cc:503] Master 127.14.237.62:40949 was elected leader, sending a full tablet report...
I20260810 04:46:08.823904 22998 raft_consensus.cc:348] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.824059 22998 raft_consensus.cc:374] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:08.824108 22998 raft_consensus.cc:729] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad0661f516a4adf977654a1bca58a5c, State: Initialized, Role: FOLLOWER
I20260810 04:46:08.824242 22998 consensus_queue.cc:260] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.824509 22998 ts_tablet_manager.cc:1428] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:46:08.824549 22957 heartbeater.cc:503] Master 127.14.237.62:40949 was elected leader, sending a full tablet report...
W20260810 04:46:08.864912 22736 tablet.cc:2369] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:46:08.920743 23000 raft_consensus.cc:482] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:08.920897 23000 raft_consensus.cc:504] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.921306 23000 leader_election.cc:288] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ad0661f516a4adf977654a1bca58a5c (127.14.237.3:46113), 101f8543727048e2b0dadea4d5e8b088 (127.14.237.1:35707)
I20260810 04:46:08.924345 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.921552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:54679 (local address 127.14.237.3:46113)
0810 04:46:08.921621 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.921623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.921633 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:08.921717 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.921719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.921731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.921785 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:08.921938 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.922398 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.923576 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.923966 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.924044 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.924053 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.924230 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.924266 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:46:08.924360 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.921497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:46113 (local address 127.14.237.2:54679)
0810 04:46:08.921573 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.921585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:08.921628 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:08.921805 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:08.921805 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:08.921807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:08.921876 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.921878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.922478 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.922479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.923477 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.923480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.924044 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.924045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.924141 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.924144 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:08.924212 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:46:08.924251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 04:46:08.924379 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.921592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:35205 (local address 127.14.237.1:35707)
0810 04:46:08.921635 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.921637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.921778 (+ 141us) server_negotiation.cc:407] Connection header received
0810 04:46:08.921841 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.921842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.921852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.921916 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:08.922059 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.922502 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.923624 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.923995 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.924093 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.924100 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.924264 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.924306 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:46:08.924427 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.921513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:35707 (local address 127.14.237.2:35205)
0810 04:46:08.921701 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.921709 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:08.921786 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:08.921919 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:08.921920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:08.921922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:08.921993 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.921994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.922623 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.922624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.923537 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:08.923540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.924093 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.924094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:08.924174 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.924177 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:08.924236 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:46:08.924276 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":126,"threads_started":1}
I20260810 04:46:08.924752 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211b6160fa734cb6b513d3138edfb893" candidate_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad0661f516a4adf977654a1bca58a5c" is_pre_election: true
I20260810 04:46:08.924813 22439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211b6160fa734cb6b513d3138edfb893" candidate_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8543727048e2b0dadea4d5e8b088" is_pre_election: true
I20260810 04:46:08.924912 22874 raft_consensus.cc:2454] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a4e25ba8dff4b6db58fff6e291c75a9 in term 0.
I20260810 04:46:08.924990 22439 raft_consensus.cc:2454] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a4e25ba8dff4b6db58fff6e291c75a9 in term 0.
I20260810 04:46:08.925138 22541 leader_election.cc:302] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad0661f516a4adf977654a1bca58a5c, 8a4e25ba8dff4b6db58fff6e291c75a9; no voters:
I20260810 04:46:08.925297 23000 raft_consensus.cc:2781] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:08.925361 23000 raft_consensus.cc:482] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:08.925377 23000 raft_consensus.cc:3037] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:08.926028 23000 raft_consensus.cc:504] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.926162 23000 leader_election.cc:288] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [CANDIDATE]: Term 1 election: Requested vote from peers 0ad0661f516a4adf977654a1bca58a5c (127.14.237.3:46113), 101f8543727048e2b0dadea4d5e8b088 (127.14.237.1:35707)
I20260810 04:46:08.926294 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211b6160fa734cb6b513d3138edfb893" candidate_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad0661f516a4adf977654a1bca58a5c"
I20260810 04:46:08.926295 22439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211b6160fa734cb6b513d3138edfb893" candidate_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101f8543727048e2b0dadea4d5e8b088"
I20260810 04:46:08.926365 22874 raft_consensus.cc:3037] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:08.926365 22439 raft_consensus.cc:3037] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:08.927263 22874 raft_consensus.cc:2454] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4e25ba8dff4b6db58fff6e291c75a9 in term 1.
I20260810 04:46:08.927263 22439 raft_consensus.cc:2454] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4e25ba8dff4b6db58fff6e291c75a9 in term 1.
I20260810 04:46:08.927433 22541 leader_election.cc:302] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad0661f516a4adf977654a1bca58a5c, 8a4e25ba8dff4b6db58fff6e291c75a9; no voters:
I20260810 04:46:08.927556 23000 raft_consensus.cc:2781] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:08.927713 23000 raft_consensus.cc:686] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 1 LEADER]: Becoming Leader. State: Replica: 8a4e25ba8dff4b6db58fff6e291c75a9, State: Running, Role: LEADER
I20260810 04:46:08.927829 23000 consensus_queue.cc:237] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } }
I20260810 04:46:08.928640 22189 catalog_manager.cc:5495] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a4e25ba8dff4b6db58fff6e291c75a9 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a4e25ba8dff4b6db58fff6e291c75a9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 } health_report { overall_health: UNKNOWN } } }
W20260810 04:46:08.954521 22511 tablet.cc:2369] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:46:08.956055 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.953597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.14.237.2:33629)
0810 04:46:08.953659 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:08.953661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:08.953670 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:08.953776 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:08.953781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:08.953812 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:08.953887 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:08.954033 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.954520 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.955314 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:08.955496 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:08.955568 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:08.955728 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:08.955824 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:08.955914 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:08.955933 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 04:46:08.955952 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:46:08.957772 22439 raft_consensus.cc:1261] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 1 FOLLOWER]: Refusing update from remote peer 8a4e25ba8dff4b6db58fff6e291c75a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:08.957773 22874 raft_consensus.cc:1261] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 1 FOLLOWER]: Refusing update from remote peer 8a4e25ba8dff4b6db58fff6e291c75a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:08.958197 23004 consensus_queue.cc:1035] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad0661f516a4adf977654a1bca58a5c" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 46113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:08.958341 23000 consensus_queue.cc:1035] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101f8543727048e2b0dadea4d5e8b088" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:08.961508 23009 mvcc.cc:205] Tried to move back new op lower bound from 7316837044048470016 to 7316837043928956928. Current Snapshot: MvccSnapshot[applied={T|T < 7316837044048470016}]
I20260810 04:46:08.963564 22187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:08.963666 22187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:08.964440 22187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29d6d773c8aa46f3899b903b3b041784: Sending DeleteTablet for 3 replicas of tablet 211b6160fa734cb6b513d3138edfb893
I20260810 04:46:08.964754 22843 tablet_service.cc:1514] Processing DeleteTablet for tablet 211b6160fa734cb6b513d3138edfb893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:08 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:39324
I20260810 04:46:08.964784 22403 tablet_service.cc:1514] Processing DeleteTablet for tablet 211b6160fa734cb6b513d3138edfb893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:08 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:39668
I20260810 04:46:08.964977 23028 tablet_replica.cc:330] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088: stopping tablet replica
I20260810 04:46:08.965062 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 22298
I20260810 04:46:08.965073 23028 raft_consensus.cc:2229] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:08.965188 23028 raft_consensus.cc:2258] T 211b6160fa734cb6b513d3138edfb893 P 101f8543727048e2b0dadea4d5e8b088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:08.968117 23027 tablet_replica.cc:330] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: stopping tablet replica
I20260810 04:46:08.968221 23027 raft_consensus.cc:2229] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:08.968329 23027 raft_consensus.cc:2258] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:08.974166 22613 tablet_service.cc:1514] Processing DeleteTablet for tablet 211b6160fa734cb6b513d3138edfb893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:08 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:40512
I20260810 04:46:08.974936 23029 tablet_replica.cc:330] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: stopping tablet replica
I20260810 04:46:08.975021 23029 raft_consensus.cc:2229] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:08.975136 23029 raft_consensus.cc:2258] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:08.976866 23029 ts_tablet_manager.cc:1905] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:08.978595 23029 ts_tablet_manager.cc:1918] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:08.978662 23029 log.cc:1196] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/211b6160fa734cb6b513d3138edfb893
I20260810 04:46:08.978940 23029 ts_tablet_manager.cc:1939] T 211b6160fa734cb6b513d3138edfb893 P 8a4e25ba8dff4b6db58fff6e291c75a9: Deleting consensus metadata
I20260810 04:46:08.979331 22168 catalog_manager.cc:4841] TS 8a4e25ba8dff4b6db58fff6e291c75a9 (127.14.237.2:33629): tablet 211b6160fa734cb6b513d3138edfb893 (table test-table [id=cac9e5cab1f64b5caebbedccbdafc9a3]) successfully deleted
I20260810 04:46:08.979712 23027 ts_tablet_manager.cc:1905] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:08.981402 23027 ts_tablet_manager.cc:1918] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:08.981463 23027 log.cc:1196] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/211b6160fa734cb6b513d3138edfb893
I20260810 04:46:08.981674 23027 ts_tablet_manager.cc:1939] T 211b6160fa734cb6b513d3138edfb893 P 0ad0661f516a4adf977654a1bca58a5c: Deleting consensus metadata
I20260810 04:46:08.982285 22166 catalog_manager.cc:4841] TS 0ad0661f516a4adf977654a1bca58a5c (127.14.237.3:46113): tablet 211b6160fa734cb6b513d3138edfb893 (table test-table [id=cac9e5cab1f64b5caebbedccbdafc9a3]) successfully deleted
W20260810 04:46:08.986388 22168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:35707 (error 108)
I20260810 04:46:08.986756 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 22517
I20260810 04:46:08.987347 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:08.986803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:35707 (local address 127.0.0.1:39672)
0810 04:46:08.987253 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:08.987298 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425}
W20260810 04:46:08.987483 22168 catalog_manager.cc:4584] TS 101f8543727048e2b0dadea4d5e8b088 (127.14.237.1:35707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 211b6160fa734cb6b513d3138edfb893: Network error: Client connection negotiation failed: client connection to 127.14.237.1:35707: connect: Connection refused (error 111)
I20260810 04:46:08.997956 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 22740
I20260810 04:46:09.008178 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 22141
2026-08-10T04:46:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2237 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T04:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:09Z Disabled control of system clock
I20260810 04:46:09.045655 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37419
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 04:46:09.127606 23042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:09.127839 23042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:09.127861 23042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:09.128933 23042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:09.128965 23042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:09.128978 23042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:09.128989 23042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:09.129001 23042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:09.130257 23042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:37419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:09.130442 23042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:09.130676 23042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:09.135537 23052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:09.135592 23042 server_base.cc:1034] running on GCE node
W20260810 04:46:09.135528 23058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:09.135517 23055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:09.136045 23042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:09.136368 23042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:09.137524 23042 hybrid_clock.cc:648] HybridClock initialized: now 1786337169137515 us; error 35 us; skew 500 ppm
I20260810 04:46:09.138787 23042 webserver.cc:460] Webserver started at http://127.14.237.62:38373/ using document root <none> and password file <none>
I20260810 04:46:09.139070 23042 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:09.139176 23042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:09.139324 23042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:09.140408 23042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "e1b1818aa7b0480c9df76505b91c4ca6"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "61b0551a1af4104aa173cf37742f6597"
server_key_iv: "bcb54f40cb835656b312a8fd0069a42c"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.140792 23042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "e1b1818aa7b0480c9df76505b91c4ca6"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "61b0551a1af4104aa173cf37742f6597"
server_key_iv: "bcb54f40cb835656b312a8fd0069a42c"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.142202 23042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:09.142925 23068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.143093 23042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:09.143187 23042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "e1b1818aa7b0480c9df76505b91c4ca6"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "61b0551a1af4104aa173cf37742f6597"
server_key_iv: "bcb54f40cb835656b312a8fd0069a42c"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.143265 23042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:09.146515 23042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:09.146780 23042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:09.146917 23042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:09.150826 23042 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:37419
I20260810 04:46:09.150858 23166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:37419 every 8 connection(s)
I20260810 04:46:09.151216 23042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:09.151898 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:09.154749 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6: Bootstrap starting.
I20260810 04:46:09.155335 23169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:09.155627 23169 log.cc:824] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:09.156291 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6: No bootstrap required, opened a new log
I20260810 04:46:09.157936 23169 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b1818aa7b0480c9df76505b91c4ca6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37419 } }
I20260810 04:46:09.158068 23169 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:09.158090 23169 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b1818aa7b0480c9df76505b91c4ca6, State: Initialized, Role: FOLLOWER
I20260810 04:46:09.158270 23169 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b1818aa7b0480c9df76505b91c4ca6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37419 } }
I20260810 04:46:09.158349 23169 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:09.158396 23169 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:09.158443 23169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:09.158975 23169 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b1818aa7b0480c9df76505b91c4ca6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37419 } }
I20260810 04:46:09.159087 23169 leader_election.cc:302] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1b1818aa7b0480c9df76505b91c4ca6; no voters:
I20260810 04:46:09.159268 23169 leader_election.cc:288] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:09.159317 23173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:09.159528 23169 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:09.159511 23173 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [term 1 LEADER]: Becoming Leader. State: Replica: e1b1818aa7b0480c9df76505b91c4ca6, State: Running, Role: LEADER
I20260810 04:46:09.159862 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 23042
I20260810 04:46:09.159972 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:09.160166 15284 external_mini_cluster.cc:1428] Setting key 4b9a7f3030de3a608b59e51d5e054fbd
I20260810 04:46:09.160277 23173 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b1818aa7b0480c9df76505b91c4ca6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37419 } }
I20260810 04:46:09.160960 23176 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1b1818aa7b0480c9df76505b91c4ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b1818aa7b0480c9df76505b91c4ca6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37419 } } }
I20260810 04:46:09.161043 23176 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:09.161473 23203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:09.161473 23177 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b1818aa7b0480c9df76505b91c4ca6. Latest consensus state: current_term: 1 leader_uuid: "e1b1818aa7b0480c9df76505b91c4ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b1818aa7b0480c9df76505b91c4ca6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37419 } } }
I20260810 04:46:09.161623 23177 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:09.162000 23203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:09.163359 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.160830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.14.237.62:37419)
0810 04:46:09.161046 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.161056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.161291 (+ 235us) server_negotiation.cc:407] Connection header received
0810 04:46:09.161486 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.161495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.161543 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.161662 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.162878 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.162881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.162882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.162886 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.162935 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.162946 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.163028 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.163104 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.163173 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260810 04:46:09.163856 23203 catalog_manager.cc:1347] Generated new cluster ID: ec917317243047c2ae4f8bf7a7e6f054
I20260810 04:46:09.163911 23203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:09.174710 23203 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:09.174865 23203 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:09.180917 23203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6: Generated new TSK 0
I20260810 04:46:09.181149 23203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:46:09.190060 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37419
--builtin_ntp_servers=127.14.237.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 04:46:09.272341 23209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:09.272545 23209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:09.272567 23209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:09.273759 23209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:09.273798 23209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:09.273835 23209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:09.275149 23209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:09.275414 23209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:09.275712 23209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:09.281473 23224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:09.281469 23228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:09.281499 23225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:09.281949 23209 server_base.cc:1034] running on GCE node
I20260810 04:46:09.282159 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:09.282403 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:09.283578 23209 hybrid_clock.cc:648] HybridClock initialized: now 1786337169283568 us; error 29 us; skew 500 ppm
I20260810 04:46:09.284879 23209 webserver.cc:460] Webserver started at http://127.14.237.1:40215/ using document root <none> and password file <none>
I20260810 04:46:09.285132 23209 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:09.285182 23209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:09.285351 23209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:09.286423 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "491b6164822f4576a603a7b6cabcfaa6"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "20398b87764a5a2d39ad81382088d547"
server_key_iv: "59751a46e0088b3e4ae81e024d102d7b"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.286788 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "491b6164822f4576a603a7b6cabcfaa6"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "20398b87764a5a2d39ad81382088d547"
server_key_iv: "59751a46e0088b3e4ae81e024d102d7b"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.288108 23209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:09.288852 23237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.289017 23209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:09.289121 23209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "491b6164822f4576a603a7b6cabcfaa6"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "20398b87764a5a2d39ad81382088d547"
server_key_iv: "59751a46e0088b3e4ae81e024d102d7b"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.289199 23209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:09.292505 23209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:09.292775 23209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:09.292908 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:09.293141 23209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:09.293478 23209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:09.293551 23209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.293586 23209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:09.293617 23209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.301339 23209 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:35375
I20260810 04:46:09.301374 23439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:35375 every 8 connection(s)
I20260810 04:46:09.301796 23209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:09.305615 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 23209
I20260810 04:46:09.305735 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:09.305910 15284 external_mini_cluster.cc:1428] Setting key 0a13a1ad5c6070071387ab120aa2ff6d
I20260810 04:46:09.306706 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.303432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:43249 (local address 127.14.237.62:37419)
0810 04:46:09.303482 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.303484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.303493 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:09.303523 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.303524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.303531 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.303583 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.306385 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.306387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.306387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.306393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.306413 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.306418 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.306503 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.306618 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.306636 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 04:46:09.307093 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37419
--builtin_ntp_servers=127.14.237.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 04:46:09.308020 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.302614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37419 (local address 127.14.237.1:43249)
0810 04:46:09.302823 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.302864 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.303390 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.303665 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.303669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.303679 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:09.306213 (+ 2534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:09.306214 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:09.306258 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:09.306293 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:09.306510 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:09.306516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:09.306519 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:09.306632 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 04:46:09.306694 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260810 04:46:09.308631 23440 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37419
I20260810 04:46:09.308732 23440 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:09.308895 23440 heartbeater.cc:511] Master 127.14.237.62:37419 requested a full tablet report, sending...
I20260810 04:46:09.309482 23100 ts_manager.cc:194] Registered new tserver with Master: 491b6164822f4576a603a7b6cabcfaa6 (127.14.237.1:35375)
W20260810 04:46:09.390399 23445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:09.390594 23445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:09.390621 23445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:09.391819 23445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:09.391876 23445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:09.391901 23445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:09.393366 23445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:09.393610 23445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:09.393913 23445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:09.398761 23456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:09.398887 23445 server_base.cc:1034] running on GCE node
W20260810 04:46:09.398777 23452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:09.398777 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:09.399271 23445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:09.399551 23445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:09.400707 23445 hybrid_clock.cc:648] HybridClock initialized: now 1786337169400696 us; error 37 us; skew 500 ppm
I20260810 04:46:09.402047 23445 webserver.cc:460] Webserver started at http://127.14.237.2:41271/ using document root <none> and password file <none>
I20260810 04:46:09.402309 23445 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:09.402380 23445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:09.402522 23445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:09.403604 23445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "37d5c906db474f759939f40d3bc058f5"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "6995cd3716068c27dc82bd618fc0daac"
server_key_iv: "db86abc280224227db33d9f32934ee0c"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.403970 23445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "37d5c906db474f759939f40d3bc058f5"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "6995cd3716068c27dc82bd618fc0daac"
server_key_iv: "db86abc280224227db33d9f32934ee0c"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.405354 23445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:09.406095 23469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.406273 23445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:09.406383 23445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "37d5c906db474f759939f40d3bc058f5"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "6995cd3716068c27dc82bd618fc0daac"
server_key_iv: "db86abc280224227db33d9f32934ee0c"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.406494 23445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:09.409857 23445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:09.410108 23445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:09.410246 23445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:09.410444 23445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:09.410740 23445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:09.410790 23445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.410828 23445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:09.410854 23445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.416617 23445 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:33643
I20260810 04:46:09.416667 23669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:33643 every 8 connection(s)
I20260810 04:46:09.416957 23445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:46:09.419566 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.417670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:59173 (local address 127.14.237.62:37419)
0810 04:46:09.417720 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.417733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.417932 (+ 199us) server_negotiation.cc:407] Connection header received
0810 04:46:09.418362 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.418363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.418369 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.418418 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.419404 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.419405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.419405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.419409 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.419422 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.419426 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.419468 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.419507 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.419517 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:46:09.419763 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.417663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37419 (local address 127.14.237.2:59173)
0810 04:46:09.417826 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.417855 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.418283 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.418466 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.418469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.418477 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:09.419283 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:09.419283 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:09.419307 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:09.419332 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:09.419466 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:09.419468 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:09.419469 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:46:09.419515 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 04:46:09.419557 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260810 04:46:09.420290 23671 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37419
I20260810 04:46:09.420400 23671 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:09.420539 23671 heartbeater.cc:511] Master 127.14.237.62:37419 requested a full tablet report, sending...
I20260810 04:46:09.420899 23100 ts_manager.cc:194] Registered new tserver with Master: 37d5c906db474f759939f40d3bc058f5 (127.14.237.2:33643)
I20260810 04:46:09.423650 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 23445
I20260810 04:46:09.423763 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:09.423944 15284 external_mini_cluster.cc:1428] Setting key 43bfe71d3c2ca60df6a8974ba5eaf086
I20260810 04:46:09.424773 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37419
--builtin_ntp_servers=127.14.237.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 04:46:09.509696 23678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:09.509886 23678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:09.509904 23678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:09.510983 23678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:09.511020 23678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:09.511044 23678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:09.512496 23678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:09.512730 23678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:09.513015 23678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:09.517765 23691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:09.517791 23688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:09.517765 23694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:09.518169 23678 server_base.cc:1034] running on GCE node
I20260810 04:46:09.518356 23678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:09.518568 23678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:09.519712 23678 hybrid_clock.cc:648] HybridClock initialized: now 1786337169519699 us; error 25 us; skew 500 ppm
I20260810 04:46:09.521090 23678 webserver.cc:460] Webserver started at http://127.14.237.3:36759/ using document root <none> and password file <none>
I20260810 04:46:09.521399 23678 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:09.521466 23678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:09.521628 23678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:09.522794 23678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "0548dede2b0c48319d0fd7f774ca2e54"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "d9d9ef866bb3b3609b37d0f16aa5ac52"
server_key_iv: "74dc24a73e9ee0bdd6f3fe489846dfda"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.523195 23678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "0548dede2b0c48319d0fd7f774ca2e54"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "d9d9ef866bb3b3609b37d0f16aa5ac52"
server_key_iv: "74dc24a73e9ee0bdd6f3fe489846dfda"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.524571 23678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:09.525333 23706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.525475 23678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:09.525560 23678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "0548dede2b0c48319d0fd7f774ca2e54"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "d9d9ef866bb3b3609b37d0f16aa5ac52"
server_key_iv: "74dc24a73e9ee0bdd6f3fe489846dfda"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.525616 23678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:09.528589 23678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:09.528836 23678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:09.528970 23678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:09.529184 23678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:09.529495 23678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:09.529543 23678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.529593 23678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:09.529619 23678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.535024 23678 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:37135
I20260810 04:46:09.535126 23902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:37135 every 8 connection(s)
I20260810 04:46:09.535399 23678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:09.538364 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.536112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:36987 (local address 127.14.237.62:37419)
0810 04:46:09.536159 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.536161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.536408 (+ 247us) server_negotiation.cc:407] Connection header received
0810 04:46:09.536895 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.536896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.536903 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.536953 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.538112 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.538113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.538114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.538119 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.538134 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.538139 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.538192 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.538278 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.538294 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:46:09.538568 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.536109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37419 (local address 127.14.237.3:36987)
0810 04:46:09.536298 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.536328 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.536782 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.537018 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.537028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.537035 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:09.537961 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:09.537962 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:09.537988 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:09.538015 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:09.538194 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:09.538199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:09.538202 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:09.538287 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 04:46:09.538347 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260810 04:46:09.539076 23905 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37419
I20260810 04:46:09.539165 23905 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:09.539279 23905 heartbeater.cc:511] Master 127.14.237.62:37419 requested a full tablet report, sending...
I20260810 04:46:09.539662 23100 ts_manager.cc:194] Registered new tserver with Master: 0548dede2b0c48319d0fd7f774ca2e54 (127.14.237.3:37135)
I20260810 04:46:09.539799 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 23678
I20260810 04:46:09.539904 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:09.540148 15284 external_mini_cluster.cc:1428] Setting key f3f3c5ac4199994ab11dfadb408f8678
I20260810 04:46:09.540963 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:46:09.543264 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.542282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.14.237.62:37419)
0810 04:46:09.542317 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.542319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.542419 (+ 100us) server_negotiation.cc:407] Connection header received
0810 04:46:09.542481 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.542482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.542485 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.542528 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.543123 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.543123 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.543124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.543127 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.543138 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.543141 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.543181 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.543211 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.543223 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:46:09.544793 23100 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:46:09.546463 23100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:09.550060 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.548386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:37135 (local address 127.0.0.1:59370)
0810 04:46:09.548732 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.548742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.548773 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.548982 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.548988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.548995 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:09.549497 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:09.549497 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:09.549513 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:09.549536 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:09.549907 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:09.549914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:09.549917 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:09.549965 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:46:09.549979 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260810 04:46:09.550288 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.548751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.14.237.1:35375)
0810 04:46:09.548941 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.548945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.548954 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:09.548980 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.548982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.548990 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.549355 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.549742 (+ 387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.549750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.549750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.549754 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.549811 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.549844 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.550152 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.550181 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.550202 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260810 04:46:09.550083 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.548380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:35375 (local address 127.0.0.1:43946)
0810 04:46:09.548552 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.548589 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.548644 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.549091 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.549092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.549095 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:09.549655 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:09.549656 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:09.549662 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:09.549675 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:09.549932 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:09.549933 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:09.549933 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 04:46:09.549969 (+ 36us) client_negotiation.cc:240] Negotiation successful
0810 04:46:09.549984 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260810 04:46:09.550410 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.548546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.14.237.3:37135)
0810 04:46:09.548751 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.548756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.548769 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:46:09.548823 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.548825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.548837 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.548973 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.549608 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.549754 (+ 146us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.549755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.549761 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.549798 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.549821 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.550284 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.550308 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.550324 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260810 04:46:09.550527 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.548399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:33643 (local address 127.0.0.1:51796)
0810 04:46:09.548936 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.548945 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.548986 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.549654 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.549655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.549658 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:09.550090 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:09.550090 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:09.550097 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:09.550109 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:09.550343 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:09.550389 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:09.550390 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:46:09.550446 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:46:09.550460 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":90,"threads_started":1}
I20260810 04:46:09.550567 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.548611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.14.237.2:33643)
0810 04:46:09.548908 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.548912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.549038 (+ 126us) server_negotiation.cc:407] Connection header received
0810 04:46:09.549067 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.549070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.549080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.549151 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.550197 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.550203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.550203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.550207 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.550240 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.550261 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.550318 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.550464 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.550483 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260810 04:46:09.551011 23325 tablet_service.cc:1467] Processing CreateTablet for tablet 53b63931f6d0475091b2e8deb5952515 (DEFAULT_TABLE table=test-table [id=49a1dda252cd4f2783905177687966fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:09.551033 23791 tablet_service.cc:1467] Processing CreateTablet for tablet 53b63931f6d0475091b2e8deb5952515 (DEFAULT_TABLE table=test-table [id=49a1dda252cd4f2783905177687966fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:09.551121 23562 tablet_service.cc:1467] Processing CreateTablet for tablet 53b63931f6d0475091b2e8deb5952515 (DEFAULT_TABLE table=test-table [id=49a1dda252cd4f2783905177687966fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:09.551308 23325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b63931f6d0475091b2e8deb5952515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:09.551397 23791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b63931f6d0475091b2e8deb5952515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:09.551402 23562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b63931f6d0475091b2e8deb5952515. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:09.554355 23929 tablet_bootstrap.cc:492] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6: Bootstrap starting.
I20260810 04:46:09.554818 23930 tablet_bootstrap.cc:492] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Bootstrap starting.
I20260810 04:46:09.554857 23931 tablet_bootstrap.cc:492] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: Bootstrap starting.
I20260810 04:46:09.555218 23929 tablet_bootstrap.cc:654] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:09.555512 23930 tablet_bootstrap.cc:654] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:09.555521 23931 tablet_bootstrap.cc:654] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:09.555568 23929 log.cc:824] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:09.555850 23931 log.cc:824] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:09.555860 23930 log.cc:824] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:09.556649 23929 tablet_bootstrap.cc:492] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6: No bootstrap required, opened a new log
I20260810 04:46:09.556655 23930 tablet_bootstrap.cc:492] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: No bootstrap required, opened a new log
I20260810 04:46:09.556705 23929 ts_tablet_manager.cc:1397] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:09.556730 23930 ts_tablet_manager.cc:1397] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:09.557667 23931 tablet_bootstrap.cc:492] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: No bootstrap required, opened a new log
I20260810 04:46:09.557747 23931 ts_tablet_manager.cc:1397] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:46:09.558883 23929 raft_consensus.cc:348] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.559074 23929 raft_consensus.cc:374] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:09.559122 23929 raft_consensus.cc:729] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491b6164822f4576a603a7b6cabcfaa6, State: Initialized, Role: FOLLOWER
I20260810 04:46:09.559244 23929 consensus_queue.cc:260] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.559510 23929 ts_tablet_manager.cc:1428] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:46:09.559473 23930 raft_consensus.cc:348] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.559651 23440 heartbeater.cc:503] Master 127.14.237.62:37419 was elected leader, sending a full tablet report...
I20260810 04:46:09.559702 23930 raft_consensus.cc:374] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:09.559756 23930 raft_consensus.cc:729] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d5c906db474f759939f40d3bc058f5, State: Initialized, Role: FOLLOWER
I20260810 04:46:09.559908 23930 consensus_queue.cc:260] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.560199 23671 heartbeater.cc:503] Master 127.14.237.62:37419 was elected leader, sending a full tablet report...
I20260810 04:46:09.560422 23931 raft_consensus.cc:348] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.560598 23931 raft_consensus.cc:374] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:09.560636 23931 raft_consensus.cc:729] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0548dede2b0c48319d0fd7f774ca2e54, State: Initialized, Role: FOLLOWER
I20260810 04:46:09.560760 23931 consensus_queue.cc:260] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.560817 23930 ts_tablet_manager.cc:1428] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 04:46:09.561028 23931 ts_tablet_manager.cc:1428] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:46:09.561084 23905 heartbeater.cc:503] Master 127.14.237.62:37419 was elected leader, sending a full tablet report...
W20260810 04:46:09.667459 23673 tablet.cc:2369] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:46:09.678328 23939 raft_consensus.cc:482] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:09.678591 23939 raft_consensus.cc:504] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.678998 23939 leader_election.cc:288] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491b6164822f4576a603a7b6cabcfaa6 (127.14.237.1:35375), 37d5c906db474f759939f40d3bc058f5 (127.14.237.2:33643)
I20260810 04:46:09.680310 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.679280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:58563 (local address 127.14.237.1:35375)
0810 04:46:09.679330 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.679332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.679347 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:46:09.679433 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.679434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.679440 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.679500 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.680084 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.680086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.680086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.680092 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.680109 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.680113 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.680172 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.680227 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.680241 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:46:09.680308 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.679239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:35375 (local address 127.14.237.3:58563)
0810 04:46:09.679303 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.679316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.679360 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.679524 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.679526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.679529 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:09.679979 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:09.679979 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:09.679986 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:09.679998 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:09.680171 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:09.680172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:09.680172 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 04:46:09.680225 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:46:09.680240 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 04:46:09.680519 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.679290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:39191 (local address 127.14.237.2:33643)
0810 04:46:09.679336 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.679338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.679576 (+ 238us) server_negotiation.cc:407] Connection header received
0810 04:46:09.679637 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.679638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.679643 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.679719 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.680247 (+ 528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.680248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.680249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.680254 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.680276 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.680280 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.680342 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.680427 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.680444 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:46:09.680727 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.679243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:33643 (local address 127.14.237.3:39191)
0810 04:46:09.679503 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.679513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.679582 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.679724 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.679724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.679726 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:09.680127 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:09.680127 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:09.680135 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:09.680153 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:09.680340 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:09.680341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:09.680342 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:46:09.680607 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 04:46:09.680623 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":63,"thread_start_us":115,"threads_started":1}
I20260810 04:46:09.680665 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b63931f6d0475091b2e8deb5952515" candidate_uuid: "0548dede2b0c48319d0fd7f774ca2e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491b6164822f4576a603a7b6cabcfaa6" is_pre_election: true
I20260810 04:46:09.680832 23341 raft_consensus.cc:2454] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0548dede2b0c48319d0fd7f774ca2e54 in term 0.
I20260810 04:46:09.681087 23595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b63931f6d0475091b2e8deb5952515" candidate_uuid: "0548dede2b0c48319d0fd7f774ca2e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d5c906db474f759939f40d3bc058f5" is_pre_election: true
I20260810 04:46:09.681205 23712 leader_election.cc:302] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0548dede2b0c48319d0fd7f774ca2e54, 491b6164822f4576a603a7b6cabcfaa6; no voters:
I20260810 04:46:09.681273 23595 raft_consensus.cc:2454] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0548dede2b0c48319d0fd7f774ca2e54 in term 0.
I20260810 04:46:09.681424 23939 raft_consensus.cc:2781] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:09.681496 23939 raft_consensus.cc:482] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:09.681521 23939 raft_consensus.cc:3037] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:09.682245 23939 raft_consensus.cc:504] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.682380 23939 leader_election.cc:288] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [CANDIDATE]: Term 1 election: Requested vote from peers 491b6164822f4576a603a7b6cabcfaa6 (127.14.237.1:35375), 37d5c906db474f759939f40d3bc058f5 (127.14.237.2:33643)
I20260810 04:46:09.682529 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b63931f6d0475091b2e8deb5952515" candidate_uuid: "0548dede2b0c48319d0fd7f774ca2e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491b6164822f4576a603a7b6cabcfaa6"
I20260810 04:46:09.682597 23595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b63931f6d0475091b2e8deb5952515" candidate_uuid: "0548dede2b0c48319d0fd7f774ca2e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d5c906db474f759939f40d3bc058f5"
I20260810 04:46:09.682619 23341 raft_consensus.cc:3037] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:09.682672 23595 raft_consensus.cc:3037] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:09.683566 23341 raft_consensus.cc:2454] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0548dede2b0c48319d0fd7f774ca2e54 in term 1.
I20260810 04:46:09.683566 23595 raft_consensus.cc:2454] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0548dede2b0c48319d0fd7f774ca2e54 in term 1.
I20260810 04:46:09.683753 23711 leader_election.cc:302] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0548dede2b0c48319d0fd7f774ca2e54, 37d5c906db474f759939f40d3bc058f5; no voters:
I20260810 04:46:09.683851 23939 raft_consensus.cc:2781] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:09.683995 23939 raft_consensus.cc:686] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 1 LEADER]: Becoming Leader. State: Replica: 0548dede2b0c48319d0fd7f774ca2e54, State: Running, Role: LEADER
I20260810 04:46:09.684098 23939 consensus_queue.cc:237] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } }
I20260810 04:46:09.684939 23100 catalog_manager.cc:5495] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0548dede2b0c48319d0fd7f774ca2e54 (127.14.237.3). New cstate: current_term: 1 leader_uuid: "0548dede2b0c48319d0fd7f774ca2e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0548dede2b0c48319d0fd7f774ca2e54" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 37135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 } health_report { overall_health: UNKNOWN } } }
I20260810 04:46:09.696187 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.695023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.14.237.3:37135)
0810 04:46:09.695070 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.695073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.695117 (+ 44us) server_negotiation.cc:407] Connection header received
0810 04:46:09.695212 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.695213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.695230 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.695307 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.695889 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.695890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.695890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.695896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.695911 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.695915 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.695996 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.696099 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.696115 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:46:09.697989 23595 raft_consensus.cc:1261] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 1 FOLLOWER]: Refusing update from remote peer 0548dede2b0c48319d0fd7f774ca2e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:09.697988 23341 raft_consensus.cc:1261] T 53b63931f6d0475091b2e8deb5952515 P 491b6164822f4576a603a7b6cabcfaa6 [term 1 FOLLOWER]: Refusing update from remote peer 0548dede2b0c48319d0fd7f774ca2e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:09.698343 23939 consensus_queue.cc:1035] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d5c906db474f759939f40d3bc058f5" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:09.698505 23941 consensus_queue.cc:1035] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491b6164822f4576a603a7b6cabcfaa6" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:09.701238 23948 mvcc.cc:205] Tried to move back new op lower bound from 7316837047080345600 to 7316837047026810880. Current Snapshot: MvccSnapshot[applied={T|T < 7316837047080345600}]
I20260810 04:46:09.701617 23947 mvcc.cc:205] Tried to move back new op lower bound from 7316837047080345600 to 7316837047026810880. Current Snapshot: MvccSnapshot[applied={T|T < 7316837047080345600}]
I20260810 04:46:09.703557 23100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:09.703644 23100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:09.704428 23100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1b1818aa7b0480c9df76505b91c4ca6: Sending DeleteTablet for 3 replicas of tablet 53b63931f6d0475091b2e8deb5952515
I20260810 04:46:09.704794 23325 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b63931f6d0475091b2e8deb5952515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:09 UTC) from {username='slave'} at 127.0.0.1:43946
I20260810 04:46:09.704798 23562 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b63931f6d0475091b2e8deb5952515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:09 UTC) from {username='slave'} at 127.0.0.1:51796
I20260810 04:46:09.704838 23791 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b63931f6d0475091b2e8deb5952515 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:09 UTC) from {username='slave'} at 127.0.0.1:59370
I20260810 04:46:09.705148 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 23209
I20260810 04:46:09.705076 23970 tablet_replica.cc:330] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: stopping tablet replica
I20260810 04:46:09.716039 23970 raft_consensus.cc:2229] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:09.716190 23970 raft_consensus.cc:2258] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:09.716185 23963 tablet_replica.cc:330] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: stopping tablet replica
I20260810 04:46:09.716281 23963 raft_consensus.cc:2229] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:09.716377 23963 raft_consensus.cc:2258] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:09.716696 23970 ts_tablet_manager.cc:1905] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:09.716775 23963 ts_tablet_manager.cc:1905] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:09.718592 23970 ts_tablet_manager.cc:1918] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:09.718592 23963 ts_tablet_manager.cc:1918] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:09.718673 23970 log.cc:1196] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/53b63931f6d0475091b2e8deb5952515
I20260810 04:46:09.718673 23963 log.cc:1196] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/53b63931f6d0475091b2e8deb5952515
I20260810 04:46:09.718976 23970 ts_tablet_manager.cc:1939] T 53b63931f6d0475091b2e8deb5952515 P 0548dede2b0c48319d0fd7f774ca2e54: Deleting consensus metadata
I20260810 04:46:09.718976 23963 ts_tablet_manager.cc:1939] T 53b63931f6d0475091b2e8deb5952515 P 37d5c906db474f759939f40d3bc058f5: Deleting consensus metadata
I20260810 04:46:09.719404 23074 catalog_manager.cc:4841] TS 0548dede2b0c48319d0fd7f774ca2e54 (127.14.237.3:37135): tablet 53b63931f6d0475091b2e8deb5952515 (table test-table [id=49a1dda252cd4f2783905177687966fe]) successfully deleted
I20260810 04:46:09.719976 23075 catalog_manager.cc:4841] TS 37d5c906db474f759939f40d3bc058f5 (127.14.237.2:33643): tablet 53b63931f6d0475091b2e8deb5952515 (table test-table [id=49a1dda252cd4f2783905177687966fe]) successfully deleted
W20260810 04:46:09.722335 23077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:35375 (error 108)
I20260810 04:46:09.722543 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 23445
I20260810 04:46:09.727702 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.722664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:35375 (local address 127.0.0.1:43956)
0810 04:46:09.727622 (+ 4958us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.727673 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4933}
W20260810 04:46:09.729544 23077 catalog_manager.cc:4584] TS 491b6164822f4576a603a7b6cabcfaa6 (127.14.237.1:35375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53b63931f6d0475091b2e8deb5952515: Network error: Client connection negotiation failed: client connection to 127.14.237.1:35375: connect: Connection refused (error 111)
I20260810 04:46:09.743688 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 23678
I20260810 04:46:09.760888 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 23042
2026-08-10T04:46:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (751 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 04:46:09.789664 15284 mini_oidc.cc:152] Starting JWKS server
I20260810 04:46:09.790813 15284 webserver.cc:460] Webserver started at https://127.0.0.1:46307/ using document root <none> and password file <none>
I20260810 04:46:09.790890 15284 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:46:09.791075 15284 webserver.cc:460] Webserver started at http://127.0.0.1:36183/ using document root <none> and password file <none>
2026-08-10T04:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:09Z Disabled control of system clock
I20260810 04:46:09.799700 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38967
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:38967
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46307/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/testchainca.cert with env {}
W20260810 04:46:09.881955 23985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:09.882148 23985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:09.882170 23985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:09.883349 23985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:09.883388 23985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:09.883401 23985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:09.883414 23985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:09.883425 23985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:46:09.883438 23985 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:46:09.883455 23985 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46307/jwks.json
I20260810 04:46:09.884958 23985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:38967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46307/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:09.885141 23985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:09.885376 23985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:09.890156 23995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:09.890307 23985 server_base.cc:1034] running on GCE node
W20260810 04:46:09.890388 23993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:09.890165 23998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:09.890700 23985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:09.891052 23985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:09.892212 23985 hybrid_clock.cc:648] HybridClock initialized: now 1786337169892203 us; error 38 us; skew 500 ppm
I20260810 04:46:09.893493 23985 webserver.cc:460] Webserver started at http://127.14.237.62:35177/ using document root <none> and password file <none>
I20260810 04:46:09.893786 23985 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:09.893867 23985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:09.893996 23985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:09.895030 23985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "a887b681cf8b4444802325dbfb13ac1d"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "a2ed9a0257b40fe614951e321d4c50b5"
server_key_iv: "df31da54af1b54debf17736e72008719"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.895391 23985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "a887b681cf8b4444802325dbfb13ac1d"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "a2ed9a0257b40fe614951e321d4c50b5"
server_key_iv: "df31da54af1b54debf17736e72008719"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.896622 23985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:09.897439 24015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:09.897631 23985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:09.897737 23985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "a887b681cf8b4444802325dbfb13ac1d"
format_stamp: "Formatted at 2026-08-10 04:46:09 on dist-test-slave-8hhm"
server_key: "a2ed9a0257b40fe614951e321d4c50b5"
server_key_iv: "df31da54af1b54debf17736e72008719"
server_key_version: "encryptionkey@0"
I20260810 04:46:09.897830 23985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:09.906867 23985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:09.907137 23985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:09.907279 23985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:09.911134 23985 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:38967
I20260810 04:46:09.911164 24101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:38967 every 8 connection(s)
I20260810 04:46:09.911536 23985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:09.912488 24106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:09.913800 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 23985
I20260810 04:46:09.913923 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:09.914096 15284 external_mini_cluster.cc:1428] Setting key 88c7b0287d9e25cc3ebf341837667a9f
I20260810 04:46:09.915263 24106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d: Bootstrap starting.
I20260810 04:46:09.915937 24106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:09.916241 24106 log.cc:824] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:09.917060 24106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d: No bootstrap required, opened a new log
I20260810 04:46:09.919406 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.914578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.14.237.62:38967)
0810 04:46:09.914766 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.914773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.914797 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:46:09.915158 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.915163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.915217 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.915326 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:09.915565 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:09.916513 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:09.917602 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:09.917992 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:09.918086 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:09.918773 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:09.918777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:09.918777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:09.918941 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:09.918970 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:09.918980 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:09.919025 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:09.919120 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 04:46:09.919174 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260810 04:46:09.919346 24106 raft_consensus.cc:348] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a887b681cf8b4444802325dbfb13ac1d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38967 } }
I20260810 04:46:09.919480 24106 raft_consensus.cc:374] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:09.919530 24106 raft_consensus.cc:729] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a887b681cf8b4444802325dbfb13ac1d, State: Initialized, Role: FOLLOWER
I20260810 04:46:09.919659 24106 consensus_queue.cc:260] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a887b681cf8b4444802325dbfb13ac1d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38967 } }
I20260810 04:46:09.919731 24106 raft_consensus.cc:388] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:09.919777 24106 raft_consensus.cc:482] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:09.919826 24106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:09.920396 24106 raft_consensus.cc:504] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a887b681cf8b4444802325dbfb13ac1d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38967 } }
I20260810 04:46:09.920401 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:46:09.920522 24106 leader_election.cc:302] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a887b681cf8b4444802325dbfb13ac1d; no voters:
I20260810 04:46:09.920702 24106 leader_election.cc:288] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:09.920773 24118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:09.920917 24118 raft_consensus.cc:686] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [term 1 LEADER]: Becoming Leader. State: Replica: a887b681cf8b4444802325dbfb13ac1d, State: Running, Role: LEADER
I20260810 04:46:09.920967 24106 sys_catalog.cc:564] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:09.921030 24118 consensus_queue.cc:237] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a887b681cf8b4444802325dbfb13ac1d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38967 } }
I20260810 04:46:09.921389 24118 sys_catalog.cc:455] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a887b681cf8b4444802325dbfb13ac1d. Latest consensus state: current_term: 1 leader_uuid: "a887b681cf8b4444802325dbfb13ac1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a887b681cf8b4444802325dbfb13ac1d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38967 } } }
I20260810 04:46:09.921468 24118 sys_catalog.cc:458] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:09.921419 24119 sys_catalog.cc:455] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a887b681cf8b4444802325dbfb13ac1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a887b681cf8b4444802325dbfb13ac1d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38967 } } }
I20260810 04:46:09.921679 24119 sys_catalog.cc:458] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d [sys.catalog]: This master's current role is: LEADER
W20260810 04:46:09.922983 24154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:46:09.923046 24154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:46:09.923128 24131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:09.923308 24131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:09.924578 24131 catalog_manager.cc:1347] Generated new cluster ID: 9e797a4924f34c6d9247b28931b98b98
I20260810 04:46:09.924628 24131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:09.930025 24131 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:09.930483 24131 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:09.935820 24131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a887b681cf8b4444802325dbfb13ac1d: Generated new TSK 0
I20260810 04:46:09.935983 24131 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 04:46:09.957154 24165 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 04:46:09.957342 24165 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 04:46:09.957530 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:09.942345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.14.237.62:38967)
0810 04:46:09.942396 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:09.942398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:09.942514 (+ 116us) server_negotiation.cc:407] Connection header received
0810 04:46:09.942617 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:09.942617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:09.942627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:09.942682 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:46:09.942856 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:09.943313 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:09.944429 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:09.944600 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:09.944646 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:09.944852 (+ 206us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:46:09.957283 (+ 12431us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 04:46:09.957454 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35576: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 04:46:09.957623 24113 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35576: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 04:46:09.958029 24163 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:09.942258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38967 (local address 127.0.0.1:35576)
0810 04:46:09.942444 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:09.942456 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:09.942523 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:09.942706 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:09.942707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:09.942709 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:46:09.942796 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:09.942798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:09.943398 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:09.943399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:09.944353 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:09.944356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:09.944668 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:09.944669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:09.944737 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:09.944758 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:46:09.957482 (+ 12724us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 04:46:09.957631 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.237.62:38967: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.237.62:38967: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 04:46:09.958449 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 23985
2026-08-10T04:46:09Z chronyd exiting
I20260810 04:46:10.002981 15284 test_util.cc:175] -----------------------------------------------
I20260810 04:46:10.003079 15284 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786337145367972-15284-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 04:46:10.003857 15284 mini_oidc.cc:152] Starting JWKS server
I20260810 04:46:10.004578 15284 webserver.cc:460] Webserver started at https://127.0.0.1:42315/ using document root <none> and password file <none>
I20260810 04:46:10.004643 15284 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:46:10.004828 15284 webserver.cc:460] Webserver started at http://127.0.0.1:38673/ using document root <none> and password file <none>
2026-08-10T04:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:10Z Disabled control of system clock
I20260810 04:46:10.014169 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:41123
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:41123
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42315/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/testchainca.cert with env {}
W20260810 04:46:10.095063 24175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:10.095268 24175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:10.095290 24175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:10.096453 24175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:10.096498 24175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:10.096510 24175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:10.096522 24175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:10.096534 24175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:46:10.096596 24175 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:46:10.096611 24175 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42315/jwks.json
I20260810 04:46:10.098457 24175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:41123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:41123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42315/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:10.098712 24175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:10.098927 24175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:10.103772 24185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:10.103801 24182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:10.104045 24175 server_base.cc:1034] running on GCE node
W20260810 04:46:10.103798 24181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:10.104346 24175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:10.104667 24175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:10.105825 24175 hybrid_clock.cc:648] HybridClock initialized: now 1786337170105809 us; error 39 us; skew 500 ppm
I20260810 04:46:10.107158 24175 webserver.cc:460] Webserver started at http://127.14.237.62:33497/ using document root <none> and password file <none>
I20260810 04:46:10.107435 24175 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:10.107509 24175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:10.107703 24175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:10.108796 24175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "502d5ad1f6a242c5becd2eaf4a4db558"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "1d6d994144d2a129544dda66ca9bc887"
server_key_iv: "e5c770d9e1aca0c7c7db9ab2b0e25ac8"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.109191 24175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "502d5ad1f6a242c5becd2eaf4a4db558"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "1d6d994144d2a129544dda66ca9bc887"
server_key_iv: "e5c770d9e1aca0c7c7db9ab2b0e25ac8"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.110666 24175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:10.111462 24195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:10.111632 24175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:10.111738 24175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "502d5ad1f6a242c5becd2eaf4a4db558"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "1d6d994144d2a129544dda66ca9bc887"
server_key_iv: "e5c770d9e1aca0c7c7db9ab2b0e25ac8"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.111832 24175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:10.123196 24175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:10.123492 24175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:10.123654 24175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:10.127700 24175 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:41123
I20260810 04:46:10.127728 24300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:41123 every 8 connection(s)
I20260810 04:46:10.128074 24175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:10.128407 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 24175
I20260810 04:46:10.128520 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:10.128724 15284 external_mini_cluster.cc:1428] Setting key 3747b36b6ef88b037e67f04ce0b1e2ad
I20260810 04:46:10.128937 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:10.132525 24301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558: Bootstrap starting.
I20260810 04:46:10.133176 24301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:10.133525 24301 log.cc:824] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:10.134231 24301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558: No bootstrap required, opened a new log
I20260810 04:46:10.135365 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:10.129607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.14.237.62:41123)
0810 04:46:10.129800 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:10.129808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:10.129841 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:46:10.130607 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:10.130617 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:10.130700 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:10.130848 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:10.131070 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.132424 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.133352 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.133842 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.133977 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.134751 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:10.134755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:10.134755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:10.134930 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:10.134962 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:10.134972 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:10.135047 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:10.135121 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 04:46:10.135197 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260810 04:46:10.136418 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:46:10.136605 24301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502d5ad1f6a242c5becd2eaf4a4db558" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41123 } }
I20260810 04:46:10.136761 24301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:10.136811 24301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502d5ad1f6a242c5becd2eaf4a4db558, State: Initialized, Role: FOLLOWER
I20260810 04:46:10.136946 24301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502d5ad1f6a242c5becd2eaf4a4db558" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41123 } }
I20260810 04:46:10.137056 24301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:10.137110 24301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:10.137181 24301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:10.137871 24301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502d5ad1f6a242c5becd2eaf4a4db558" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41123 } }
I20260810 04:46:10.137987 24301 leader_election.cc:302] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502d5ad1f6a242c5becd2eaf4a4db558; no voters:
I20260810 04:46:10.138161 24301 leader_election.cc:288] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:10.138233 24307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:10.138374 24307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [term 1 LEADER]: Becoming Leader. State: Replica: 502d5ad1f6a242c5becd2eaf4a4db558, State: Running, Role: LEADER
I20260810 04:46:10.138406 24301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:10.138463 24307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502d5ad1f6a242c5becd2eaf4a4db558" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41123 } }
I20260810 04:46:10.138818 24311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502d5ad1f6a242c5becd2eaf4a4db558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502d5ad1f6a242c5becd2eaf4a4db558" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41123 } } }
I20260810 04:46:10.138960 24311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:10.138818 24312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502d5ad1f6a242c5becd2eaf4a4db558. Latest consensus state: current_term: 1 leader_uuid: "502d5ad1f6a242c5becd2eaf4a4db558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502d5ad1f6a242c5becd2eaf4a4db558" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41123 } } }
I20260810 04:46:10.139093 24312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558 [sys.catalog]: This master's current role is: LEADER
W20260810 04:46:10.140311 24333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:46:10.140362 24333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:46:10.140429 24318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:10.140606 24318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:10.142024 24318 catalog_manager.cc:1347] Generated new cluster ID: fe19eba737ed42a6a05492374dce7141
I20260810 04:46:10.142082 24318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:10.165570 24318 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:10.166280 24318 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:10.181257 24318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502d5ad1f6a242c5becd2eaf4a4db558: Generated new TSK 0
I20260810 04:46:10.181458 24318 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 04:46:10.212654 24350 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 04:46:10.212745 24350 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 04:46:10.213059 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:10.191881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.14.237.62:41123)
0810 04:46:10.191950 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:10.191952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:10.192065 (+ 113us) server_negotiation.cc:407] Connection header received
0810 04:46:10.192146 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:10.192147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:10.192160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:10.192240 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:46:10.192381 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.192897 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.193754 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.193986 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.194064 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.194221 (+ 157us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:46:10.212807 (+ 18586us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 04:46:10.212996 (+ 189us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 04:46:10.213136 24304 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 04:46:10.213196 24349 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:10.191808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:41123 (local address 127.0.0.1:44572)
0810 04:46:10.191991 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:10.192006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:10.192075 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:10.192239 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:10.192240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:10.192242 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:46:10.192312 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:10.192314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.192998 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.192999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:10.193668 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:10.193670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.194067 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.194068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:10.194125 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.194130 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:46:10.212952 (+ 18822us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 04:46:10.213090 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.237.62:41123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":119,"threads_started":1}
I20260810 04:46:10.213654 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 24175
2026-08-10T04:46:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (255 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 04:46:10.259776 15284 mini_oidc.cc:152] Starting JWKS server
I20260810 04:46:10.260412 15284 webserver.cc:460] Webserver started at https://127.0.0.1:34665/ using document root <none> and password file <none>
I20260810 04:46:10.260452 15284 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:46:10.260613 15284 webserver.cc:460] Webserver started at http://127.0.0.1:34351/ using document root <none> and password file <none>
2026-08-10T04:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:10Z Disabled control of system clock
I20260810 04:46:10.269156 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:36787
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:36787
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34665/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 04:46:10.349838 24359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:10.350050 24359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:10.350070 24359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:10.351106 24359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:10.351140 24359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:10.351152 24359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:10.351164 24359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:10.351176 24359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:46:10.351187 24359 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:46:10.351199 24359 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34665/jwks.json
I20260810 04:46:10.352479 24359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:36787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:36787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34665/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:10.352715 24359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:10.352986 24359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:10.358031 24372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:10.358095 24375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:10.358217 24371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:10.358116 24359 server_base.cc:1034] running on GCE node
I20260810 04:46:10.358500 24359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:10.358736 24359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:10.359869 24359 hybrid_clock.cc:648] HybridClock initialized: now 1786337170359851 us; error 36 us; skew 500 ppm
I20260810 04:46:10.361104 24359 webserver.cc:460] Webserver started at http://127.14.237.62:34721/ using document root <none> and password file <none>
I20260810 04:46:10.361394 24359 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:10.361444 24359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:10.361568 24359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:10.362690 24359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "3c36ad0e6f1d4245b840ca6262ec7063"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "3870685462b7828280898ca8226a87ff"
server_key_iv: "c0ea713ba539549c56adbd1b7a486568"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.363129 24359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "3c36ad0e6f1d4245b840ca6262ec7063"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "3870685462b7828280898ca8226a87ff"
server_key_iv: "c0ea713ba539549c56adbd1b7a486568"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.364475 24359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:10.365298 24387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:10.365454 24359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:10.365545 24359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "3c36ad0e6f1d4245b840ca6262ec7063"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "3870685462b7828280898ca8226a87ff"
server_key_iv: "c0ea713ba539549c56adbd1b7a486568"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.365607 24359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:10.376507 24359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:10.376749 24359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:10.376847 24359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:10.380514 24359 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:36787
I20260810 04:46:10.380559 24478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:36787 every 8 connection(s)
I20260810 04:46:10.380890 24359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:10.381604 24479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:10.383296 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 24359
I20260810 04:46:10.383417 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:10.383634 15284 external_mini_cluster.cc:1428] Setting key 125a427e489da8a8aaa3a6820840add5
I20260810 04:46:10.384866 24479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063: Bootstrap starting.
I20260810 04:46:10.385471 24479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:10.385928 24479 log.cc:824] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:10.386893 24479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063: No bootstrap required, opened a new log
I20260810 04:46:10.389245 24479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36ad0e6f1d4245b840ca6262ec7063" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36787 } }
I20260810 04:46:10.389387 24479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:10.389410 24479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c36ad0e6f1d4245b840ca6262ec7063, State: Initialized, Role: FOLLOWER
I20260810 04:46:10.389410 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:10.384159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.14.237.62:36787)
0810 04:46:10.384386 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:10.384395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:10.384429 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:46:10.384828 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:10.384834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:10.384890 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:10.385048 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:10.385335 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.386396 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.387550 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.387955 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.388038 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.388733 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:10.388737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:10.388737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:10.388883 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:10.388913 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:10.388922 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:10.388986 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:10.389083 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:46:10.389189 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260810 04:46:10.389513 24479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36ad0e6f1d4245b840ca6262ec7063" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36787 } }
I20260810 04:46:10.389599 24479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:10.389628 24479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:10.389693 24479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:10.390259 24479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36ad0e6f1d4245b840ca6262ec7063" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36787 } }
I20260810 04:46:10.390372 24479 leader_election.cc:302] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c36ad0e6f1d4245b840ca6262ec7063; no voters:
I20260810 04:46:10.390450 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:46:10.390559 24479 leader_election.cc:288] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:10.390576 24488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:10.390750 24488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [term 1 LEADER]: Becoming Leader. State: Replica: 3c36ad0e6f1d4245b840ca6262ec7063, State: Running, Role: LEADER
I20260810 04:46:10.390803 24479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:10.390853 24488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36ad0e6f1d4245b840ca6262ec7063" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36787 } }
I20260810 04:46:10.391131 24488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c36ad0e6f1d4245b840ca6262ec7063. Latest consensus state: current_term: 1 leader_uuid: "3c36ad0e6f1d4245b840ca6262ec7063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36ad0e6f1d4245b840ca6262ec7063" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36787 } } }
I20260810 04:46:10.391225 24488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:10.391141 24494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c36ad0e6f1d4245b840ca6262ec7063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36ad0e6f1d4245b840ca6262ec7063" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 36787 } } }
I20260810 04:46:10.391386 24494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063 [sys.catalog]: This master's current role is: LEADER
W20260810 04:46:10.392513 24520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:46:10.392573 24520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:46:10.392635 24502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:10.392966 24502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:10.394457 24502 catalog_manager.cc:1347] Generated new cluster ID: 87af55bc106443a390f0bab39d486e39
I20260810 04:46:10.394505 24502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:10.406659 24502 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:10.407351 24502 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:10.412791 24502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c36ad0e6f1d4245b840ca6262ec7063: Generated new TSK 0
I20260810 04:46:10.412962 24502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:46:10.423022 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:10.419860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.14.237.62:36787)
0810 04:46:10.419931 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:10.419933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:10.420043 (+ 110us) server_negotiation.cc:407] Connection header received
0810 04:46:10.420146 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:10.420147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:10.420157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:10.420210 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:46:10.420343 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.420826 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.421668 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.421855 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.421920 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.422080 (+ 160us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:46:10.422810 (+ 730us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 04:46:10.422951 (+ 141us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50736: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 04:46:10.423075 24486 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50736: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 04:46:10.423106 24541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:10.419798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:36787 (local address 127.0.0.1:50736)
0810 04:46:10.419986 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:10.419995 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:10.420058 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:10.420223 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:10.420224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:10.420226 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:46:10.420276 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:10.420278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.420908 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.420909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:10.421579 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:10.421581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.421925 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.421926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:10.421985 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.421990 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:46:10.422908 (+ 918us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 04:46:10.422997 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.237.62:36787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260810 04:46:10.423671 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 24359
E20260810 04:46:10.446262 24546 webserver.cc:533] Webserver: sslize failed:
E20260810 04:46:10.446346 24546 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T04:46:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T04:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:10Z Disabled control of system clock
I20260810 04:46:10.481922 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:42269
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:42269
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/testchainca.cert with env {}
W20260810 04:46:10.562119 24558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:10.562332 24558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:10.562354 24558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:10.563477 24558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:10.563512 24558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:10.563525 24558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:10.563539 24558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:10.563550 24558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:46:10.563562 24558 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:46:10.563573 24558 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 04:46:10.564990 24558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:42269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:10.565181 24558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:10.565407 24558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:10.570078 24565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:10.570101 24571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:10.570314 24558 server_base.cc:1034] running on GCE node
W20260810 04:46:10.570070 24567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:10.570657 24558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:10.570947 24558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:10.572113 24558 hybrid_clock.cc:648] HybridClock initialized: now 1786337170572084 us; error 39 us; skew 500 ppm
I20260810 04:46:10.573469 24558 webserver.cc:460] Webserver started at http://127.14.237.62:46423/ using document root <none> and password file <none>
I20260810 04:46:10.573729 24558 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:10.573777 24558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:10.573956 24558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:10.575035 24558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "7031103c43524bf8af37cc42237ebde0"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "5a8990630818817a5c588dc54b501d09"
server_key_iv: "73c7addb6bf036603fd014339ecca149"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.575402 24558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "7031103c43524bf8af37cc42237ebde0"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "5a8990630818817a5c588dc54b501d09"
server_key_iv: "73c7addb6bf036603fd014339ecca149"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.576707 24558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:10.577467 24580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:10.577663 24558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:10.577755 24558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "7031103c43524bf8af37cc42237ebde0"
format_stamp: "Formatted at 2026-08-10 04:46:10 on dist-test-slave-8hhm"
server_key: "5a8990630818817a5c588dc54b501d09"
server_key_iv: "73c7addb6bf036603fd014339ecca149"
server_key_version: "encryptionkey@0"
I20260810 04:46:10.577828 24558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:10.585655 24558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:10.585908 24558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:10.586004 24558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:10.589943 24558 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:42269
I20260810 04:46:10.589974 24685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:42269 every 8 connection(s)
I20260810 04:46:10.590276 24558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:10.591169 24690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:10.593628 24690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0: Bootstrap starting.
I20260810 04:46:10.594208 24690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:10.594501 24690 log.cc:824] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:10.595149 24690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0: No bootstrap required, opened a new log
I20260810 04:46:10.596304 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 24558
I20260810 04:46:10.596417 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:10.596608 15284 external_mini_cluster.cc:1428] Setting key 70a3ba492232ab507672a7ef617a3723
I20260810 04:46:10.597150 24690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7031103c43524bf8af37cc42237ebde0" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42269 } }
I20260810 04:46:10.597371 24690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:10.597425 24690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7031103c43524bf8af37cc42237ebde0, State: Initialized, Role: FOLLOWER
I20260810 04:46:10.597572 24690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7031103c43524bf8af37cc42237ebde0" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42269 } }
I20260810 04:46:10.597661 24690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:10.597702 24690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:10.597748 24690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:10.598289 24690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7031103c43524bf8af37cc42237ebde0" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42269 } }
I20260810 04:46:10.598404 24690 leader_election.cc:302] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7031103c43524bf8af37cc42237ebde0; no voters:
I20260810 04:46:10.598570 24690 leader_election.cc:288] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:10.598610 24698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:10.598717 24698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [term 1 LEADER]: Becoming Leader. State: Replica: 7031103c43524bf8af37cc42237ebde0, State: Running, Role: LEADER
I20260810 04:46:10.598832 24690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:10.598805 24698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7031103c43524bf8af37cc42237ebde0" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42269 } }
I20260810 04:46:10.599169 24698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7031103c43524bf8af37cc42237ebde0. Latest consensus state: current_term: 1 leader_uuid: "7031103c43524bf8af37cc42237ebde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7031103c43524bf8af37cc42237ebde0" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42269 } } }
I20260810 04:46:10.599251 24698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:10.599179 24701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7031103c43524bf8af37cc42237ebde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7031103c43524bf8af37cc42237ebde0" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42269 } } }
I20260810 04:46:10.599396 24701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:10.599473 24707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:10.600104 24707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:10.601809 24707 catalog_manager.cc:1347] Generated new cluster ID: 4b23e8a864694d56b36503e6571c3d11
I20260810 04:46:10.601861 24707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:10.602994 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:10.597111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.14.237.62:42269)
0810 04:46:10.597359 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:10.597369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:10.597401 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:46:10.597522 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:10.597529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:10.597587 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:10.597721 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:10.597972 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.599530 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.600782 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.601450 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.601581 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.602251 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:10.602255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:10.602256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:10.602478 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:10.602519 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:10.602533 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:10.602614 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:10.602687 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 04:46:10.602771 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":118,"threads_started":1}
I20260810 04:46:10.603904 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:46:10.610380 24707 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:10.610836 24707 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:10.616544 24707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7031103c43524bf8af37cc42237ebde0: Generated new TSK 0
I20260810 04:46:10.616703 24707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:46:10.664047 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:10.621278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.14.237.62:42269)
0810 04:46:10.621328 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:10.621330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:10.621478 (+ 148us) server_negotiation.cc:407] Connection header received
0810 04:46:10.621550 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:10.621551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:10.621561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:10.621625 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:46:10.621778 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.622251 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.623403 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.623580 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.623635 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.623812 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:46:10.663818 (+ 40006us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 04:46:10.663975 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33766: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":26}
W20260810 04:46:10.664148 24697 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33766: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 04:46:10.664259 24753 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:10.621176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:42269 (local address 127.0.0.1:33766)
0810 04:46:10.621405 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:10.621416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:10.621482 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:10.621636 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:10.621637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:10.621640 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:46:10.621701 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:10.621703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.622336 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.622337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:10.623309 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:10.623312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:10.623653 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:10.623654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:10.623714 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:10.623719 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:46:10.663975 (+ 40256us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 04:46:10.664149 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.237.62:42269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":112,"threads_started":1}
I20260810 04:46:10.664734 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 24558
2026-08-10T04:46:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (217 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:10 dist-test-slave-8hhm krb5kdc[24772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:10 dist-test-slave-8hhm krb5kdc[24772](info): set up 2 sockets
Aug 10 04:46:10 dist-test-slave-8hhm krb5kdc[24772](info): commencing operation
krb5kdc: starting...
W20260810 04:46:12.116935 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:12.237201 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:45439
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:45439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:12.318651 24800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:12.318876 24800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:12.318910 24800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:12.320011 24800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:12.320076 24800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:12.320099 24800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:12.320118 24800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:12.320147 24800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:12.321719 24800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:45439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:45439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:12.321973 24800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:12.322186 24800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:12.327149 24813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:12.327198 24800 server_base.cc:1034] running on GCE node
W20260810 04:46:12.327171 24812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:12.327171 24817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:12.327561 24800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:12.327814 24800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:12.328948 24800 hybrid_clock.cc:648] HybridClock initialized: now 1786337172328936 us; error 27 us; skew 500 ppm
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.331465 24800 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:12.331909 24800 webserver.cc:460] Webserver started at http://127.14.237.62:35591/ using document root <none> and password file <none>
I20260810 04:46:12.332159 24800 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:12.332207 24800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:12.332358 24800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:12.333482 24800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "032bf3055451488683fc84d8a6c5be05"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "b4725938798d81e12de750ce7aef0029"
server_key_iv: "fcc6d8fc320bb3d1786896c91e5a2abc"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.333851 24800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "032bf3055451488683fc84d8a6c5be05"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "b4725938798d81e12de750ce7aef0029"
server_key_iv: "fcc6d8fc320bb3d1786896c91e5a2abc"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.335250 24800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:12.335933 24835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.336104 24800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:12.336197 24800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "032bf3055451488683fc84d8a6c5be05"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "b4725938798d81e12de750ce7aef0029"
server_key_iv: "fcc6d8fc320bb3d1786896c91e5a2abc"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.336263 24800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:12.349489 24800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:12.351723 24800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:12.351850 24800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:12.355868 24800 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:45439
I20260810 04:46:12.355916 24935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:45439 every 8 connection(s)
I20260810 04:46:12.356278 24800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:12.357177 24936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:12.359478 24936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05: Bootstrap starting.
I20260810 04:46:12.360016 24936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:12.360280 24936 log.cc:824] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:12.360930 24936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05: No bootstrap required, opened a new log
I20260810 04:46:12.362280 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 24800
I20260810 04:46:12.362406 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:12.362624 15284 external_mini_cluster.cc:1428] Setting key 9e58731253a7abcb07cd7ae450c52a03
I20260810 04:46:12.363263 24936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032bf3055451488683fc84d8a6c5be05" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45439 } }
I20260810 04:46:12.363466 24936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:12.363520 24936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032bf3055451488683fc84d8a6c5be05, State: Initialized, Role: FOLLOWER
I20260810 04:46:12.363659 24936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032bf3055451488683fc84d8a6c5be05" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45439 } }
I20260810 04:46:12.363765 24936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:12.363802 24936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:12.363867 24936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:12.364970 24936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032bf3055451488683fc84d8a6c5be05" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45439 } }
I20260810 04:46:12.365070 24936 leader_election.cc:302] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032bf3055451488683fc84d8a6c5be05; no voters:
I20260810 04:46:12.365286 24936 leader_election.cc:288] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:12.365315 24947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:12.365424 24947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [term 1 LEADER]: Becoming Leader. State: Replica: 032bf3055451488683fc84d8a6c5be05, State: Running, Role: LEADER
I20260810 04:46:12.365509 24947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032bf3055451488683fc84d8a6c5be05" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45439 } }
I20260810 04:46:12.365558 24936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:12.365878 24947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032bf3055451488683fc84d8a6c5be05. Latest consensus state: current_term: 1 leader_uuid: "032bf3055451488683fc84d8a6c5be05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032bf3055451488683fc84d8a6c5be05" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45439 } } }
I20260810 04:46:12.365954 24947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:12.365890 24948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032bf3055451488683fc84d8a6c5be05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032bf3055451488683fc84d8a6c5be05" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45439 } } }
I20260810 04:46:12.366014 24948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:12.366161 24954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:12.366796 24954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:12.368652 24954 catalog_manager.cc:1347] Generated new cluster ID: b0cb6a7b1a1742eeb2e44f2d5aa04e19
I20260810 04:46:12.368714 24954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:12.374368 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.363188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.14.237.62:45439)
0810 04:46:12.363372 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.363380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.363406 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:46:12.363522 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.363529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.363601 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.363738 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.364619 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.365868 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.366995 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.367579 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.367693 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.370657 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:12.370662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:12.370662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:12.370674 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:12.372639 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.373060 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.373063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.373063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.373187 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.373420 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.373421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.373421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.373714 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:12.373858 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:12.374038 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 04:46:12.374170 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260810 04:46:12.378115 24954 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:12.378595 24954 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:12.383562 24954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032bf3055451488683fc84d8a6c5be05: Generated new TSK 0
I20260810 04:46:12.383741 24954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:46:12.410247 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45439
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:12.492059 24997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:12.492274 24997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:12.492297 24997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:12.493485 24997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:12.493525 24997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:12.493551 24997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:12.495085 24997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:12.495291 24997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:12.495532 24997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:12.500398 25009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:12.500415 25006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:12.500406 25005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:12.500715 24997 server_base.cc:1034] running on GCE node
I20260810 04:46:12.500886 24997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:12.501066 24997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:12.502192 24997 hybrid_clock.cc:648] HybridClock initialized: now 1786337172502175 us; error 33 us; skew 500 ppm
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.504603 24997 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:46:12.505100 24997 webserver.cc:460] Webserver started at http://127.14.237.1:41729/ using document root <none> and password file <none>
I20260810 04:46:12.505332 24997 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:12.505378 24997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:12.505470 24997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:12.506438 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "52cc0cd862c74cee828516215d5c32cd"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "f91bfbc7e4b1e492a15d050a23526df1"
server_key_iv: "4954c0cdf1f8b9a60efcc7c9f63c3ded"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.506742 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "52cc0cd862c74cee828516215d5c32cd"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "f91bfbc7e4b1e492a15d050a23526df1"
server_key_iv: "4954c0cdf1f8b9a60efcc7c9f63c3ded"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.507990 24997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:12.508641 25020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.508812 24997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:12.508893 24997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "52cc0cd862c74cee828516215d5c32cd"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "f91bfbc7e4b1e492a15d050a23526df1"
server_key_iv: "4954c0cdf1f8b9a60efcc7c9f63c3ded"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.508947 24997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:12.527751 24997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:12.529701 24997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:12.529820 24997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:12.530037 24997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:12.530352 24997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:12.530386 24997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.530431 24997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:12.530454 24997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.536608 24997 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:41503
I20260810 04:46:12.536681 25244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:41503 every 8 connection(s)
I20260810 04:46:12.537001 24997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:12.545697 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 24997
I20260810 04:46:12.545783 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:12.546005 15284 external_mini_cluster.cc:1428] Setting key d331d1edce9bceb88b772f20097847db
I20260810 04:46:12.546345 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.537775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:58727 (local address 127.14.237.62:45439)
0810 04:46:12.537848 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.537850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.538063 (+ 213us) server_negotiation.cc:407] Connection header received
0810 04:46:12.539090 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.539091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.539109 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.539155 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.539734 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.540299 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.541317 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.541521 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.541603 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.543626 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:12.543628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:12.543628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:12.543639 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:12.544860 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.545305 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.545306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.545307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.545363 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.545609 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.545611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.545611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.545722 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:12.545798 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:12.546150 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 04:46:12.546254 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:46:12.546569 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.537765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:45439 (local address 127.14.237.1:58727)
0810 04:46:12.537944 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:12.537980 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:12.538992 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:12.539220 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:12.539223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:12.539470 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:12.539645 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:12.539649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.540392 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.540393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:12.541177 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:12.541179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.541604 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.541605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:12.541737 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.542176 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:12.542184 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:12.543529 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:12.544955 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:12.544961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:12.544963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:12.545175 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:12.545438 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:12.545439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:12.545439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:12.545541 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:12.545839 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:12.545847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:12.545979 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 04:46:12.546122 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 04:46:12.546304 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260810 04:46:12.547125 25245 heartbeater.cc:347] Connected to a master server at 127.14.237.62:45439
I20260810 04:46:12.547222 25245 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:12.547366 25245 heartbeater.cc:511] Master 127.14.237.62:45439 requested a full tablet report, sending...
I20260810 04:46:12.547859 24872 ts_manager.cc:194] Registered new tserver with Master: 52cc0cd862c74cee828516215d5c32cd (127.14.237.1:41503)
I20260810 04:46:12.548425 24872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:58727
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:46:12.571245 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45439
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:12.653180 25254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:12.653450 25254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:12.653488 25254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:12.654687 25254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:12.654758 25254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:12.654803 25254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:12.656319 25254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:12.656599 25254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:12.656833 25254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:12.661945 25273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:12.661960 25267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:12.661960 25265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:12.662180 25254 server_base.cc:1034] running on GCE node
I20260810 04:46:12.662380 25254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:12.662608 25254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:12.663765 25254 hybrid_clock.cc:648] HybridClock initialized: now 1786337172663750 us; error 26 us; skew 500 ppm
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.666378 25254 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:46:12.666864 25254 webserver.cc:460] Webserver started at http://127.14.237.2:46859/ using document root <none> and password file <none>
I20260810 04:46:12.667095 25254 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:12.667145 25254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:12.667277 25254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:12.668282 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "25f099b17f704536abe86052baa43bba"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "de6c7134c349e40ad4d07a468d95f507"
server_key_iv: "51d14fcdc4ebeb7ae05cc9826a2434fa"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.668641 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "25f099b17f704536abe86052baa43bba"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "de6c7134c349e40ad4d07a468d95f507"
server_key_iv: "51d14fcdc4ebeb7ae05cc9826a2434fa"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.670017 25254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:12.670768 25284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.670936 25254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:12.671027 25254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "25f099b17f704536abe86052baa43bba"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "de6c7134c349e40ad4d07a468d95f507"
server_key_iv: "51d14fcdc4ebeb7ae05cc9826a2434fa"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.671099 25254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:12.688982 25254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:12.690837 25254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:12.691020 25254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:12.691249 25254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:12.691551 25254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:12.691586 25254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.691648 25254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:12.691677 25254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.697615 25254 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:43809
I20260810 04:46:12.697654 25506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:43809 every 8 connection(s)
I20260810 04:46:12.697975 25254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:12.706517 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 25254
I20260810 04:46:12.706636 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:12.706887 15284 external_mini_cluster.cc:1428] Setting key f4465b1ee963ce20fefa506ca7bfdf2d
I20260810 04:46:12.707541 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.698689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:46595 (local address 127.14.237.62:45439)
0810 04:46:12.698737 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.698740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.698923 (+ 183us) server_negotiation.cc:407] Connection header received
0810 04:46:12.699924 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.699926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.699939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.699993 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.700609 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.701093 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.702155 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.702355 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.702443 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.704693 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:12.704694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:12.704707 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:12.704714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:12.706005 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.706439 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.706441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.706441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.706514 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.706827 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.706828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.706828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.706972 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:12.707046 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:12.707368 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 04:46:12.707452 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:46:12.707715 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.698654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:45439 (local address 127.14.237.2:46595)
0810 04:46:12.698800 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:12.698841 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:12.699812 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:12.700065 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:12.700069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:12.700325 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:12.700510 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:12.700515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.701192 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.701193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:12.702054 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:12.702056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.702449 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.702451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:12.702585 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.703150 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:12.703159 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:12.704603 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:12.706090 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:12.706096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:12.706098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:12.706344 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:12.706610 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:12.706611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:12.706612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:12.706754 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:12.707035 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:12.707040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:12.707247 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 04:46:12.707333 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 04:46:12.707453 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260810 04:46:12.708218 25507 heartbeater.cc:347] Connected to a master server at 127.14.237.62:45439
I20260810 04:46:12.708335 25507 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:12.708559 25507 heartbeater.cc:511] Master 127.14.237.62:45439 requested a full tablet report, sending...
I20260810 04:46:12.708935 24872 ts_manager.cc:194] Registered new tserver with Master: 25f099b17f704536abe86052baa43bba (127.14.237.2:43809)
I20260810 04:46:12.709383 24872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:46595
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:46:12.731918 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45439
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:12.811980 25513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:12.812189 25513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:12.812212 25513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:12.813405 25513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:12.813446 25513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:12.813472 25513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:12.814956 25513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:12.815220 25513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:12.815461 25513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:12.820556 25530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:12.820572 25529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:12.820590 25513 server_base.cc:1034] running on GCE node
W20260810 04:46:12.820572 25537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:12.821012 25513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:12.821313 25513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:12.822484 25513 hybrid_clock.cc:648] HybridClock initialized: now 1786337172822469 us; error 40 us; skew 500 ppm
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.825176 25513 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:46:12.825585 25513 webserver.cc:460] Webserver started at http://127.14.237.3:33165/ using document root <none> and password file <none>
I20260810 04:46:12.825796 25513 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:12.825860 25513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:12.826006 25513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:12.827091 25513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "d86ce94a366c456bba5a7c3b937fcecd"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "3375362d9decc7fd9a0aae334097f3f0"
server_key_iv: "e0c08ea6b47c6ddf18f6fc1493627961"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.827451 25513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "d86ce94a366c456bba5a7c3b937fcecd"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "3375362d9decc7fd9a0aae334097f3f0"
server_key_iv: "e0c08ea6b47c6ddf18f6fc1493627961"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.828738 25513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:12.829511 25552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.829679 25513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:12.829792 25513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "d86ce94a366c456bba5a7c3b937fcecd"
format_stamp: "Formatted at 2026-08-10 04:46:12 on dist-test-slave-8hhm"
server_key: "3375362d9decc7fd9a0aae334097f3f0"
server_key_iv: "e0c08ea6b47c6ddf18f6fc1493627961"
server_key_version: "encryptionkey@0"
I20260810 04:46:12.829873 25513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:12.845194 25513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:12.847577 25513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:12.847739 25513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:12.847981 25513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:12.848295 25513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:12.848345 25513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.848387 25513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:12.848412 25513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:12.854204 25513 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:35565
I20260810 04:46:12.854266 25746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:35565 every 8 connection(s)
I20260810 04:46:12.854550 25513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:12.856739 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 25513
I20260810 04:46:12.856812 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:12.856995 15284 external_mini_cluster.cc:1428] Setting key 195f1c07b7c6edd7b02084196abdd9da
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:12.863767 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.855292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:40559 (local address 127.14.237.62:45439)
0810 04:46:12.855338 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.855341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.855561 (+ 220us) server_negotiation.cc:407] Connection header received
0810 04:46:12.856498 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.856499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.856511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.856569 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.857392 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.857900 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.859004 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.859181 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.859262 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.861235 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:12.861237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:12.861237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:12.861244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:12.862414 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.862813 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.862814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.862815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.862877 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.863134 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.863135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.863136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.863255 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:12.863329 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:12.863597 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 04:46:12.863678 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:46:12.863953 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.855258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:45439 (local address 127.14.237.3:40559)
0810 04:46:12.855444 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:12.855478 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:12.856388 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:12.856657 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:12.856661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:12.856981 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:12.857311 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:12.857318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.857993 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.857994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:12.858921 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:12.858923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.859272 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.859273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:12.859392 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.859780 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:12.859787 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:12.861103 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:12.862499 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:12.862506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:12.862508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:12.862736 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:12.862944 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:12.862946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:12.862946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:12.863060 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:12.863320 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:12.863327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:12.863460 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:46:12.863560 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 04:46:12.863713 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260810 04:46:12.864421 25747 heartbeater.cc:347] Connected to a master server at 127.14.237.62:45439
I20260810 04:46:12.864518 25747 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:12.864693 25747 heartbeater.cc:511] Master 127.14.237.62:45439 requested a full tablet report, sending...
I20260810 04:46:12.865100 24872 ts_manager.cc:194] Registered new tserver with Master: d86ce94a366c456bba5a7c3b937fcecd (127.14.237.3:35565)
I20260810 04:46:12.865504 24872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:40559
I20260810 04:46:12.868680 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/test-admin.keytab.
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.885833 15284 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 04:46:12.888046 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:12.886956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.14.237.1:41503)
0810 04:46:12.887158 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.887164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.887209 (+ 45us) server_negotiation.cc:407] Connection header received
0810 04:46:12.887313 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.887331 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.887349 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.887434 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.887943 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33624: BlockingRecv error: recv got EOF from 127.0.0.1:33624 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
W20260810 04:46:12.888629 15284 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5cc' of type FILE
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.889923 15284 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.890833 15284 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:46:12.898424 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.891378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.14.237.1:41503)
0810 04:46:12.891431 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.891433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.891500 (+ 67us) server_negotiation.cc:407] Connection header received
0810 04:46:12.891544 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.891545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.891556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.891601 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.891981 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.892719 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.893475 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.893792 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.893845 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.895882 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:12.895890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:12.895890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:12.895897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:12.897434 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.897757 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.897760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.897760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.897817 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.897974 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.897975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.897975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.898133 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:12.898241 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:12.898299 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 04:46:12.898356 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 04:46:12.900313 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:12.899596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.14.237.1:41503)
0810 04:46:12.899631 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.899633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.899723 (+ 90us) server_negotiation.cc:407] Connection header received
0810 04:46:12.899791 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.899792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.899802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.899868 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.900260 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33630: BlockingRecv error: recv got EOF from 127.0.0.1:33630 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 04:46:12.900424 15284 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5cc' of type FILE
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.901383 15284 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.902171 15284 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:46:12.908815 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.902644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.14.237.1:41503)
0810 04:46:12.902706 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.902708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.902752 (+ 44us) server_negotiation.cc:407] Connection header received
0810 04:46:12.902789 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.902789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.902797 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.902844 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.903205 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.903638 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.904282 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.904460 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.904514 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.906672 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:12.906674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:12.906674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:12.906681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:12.907986 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.908283 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.908284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.908284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.908331 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.908479 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.908479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.908479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.908556 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:12.908619 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:12.908703 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 04:46:12.908765 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 04:46:12.910279 25765 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:12.909788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.14.237.1:41503)
0810 04:46:12.909837 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.909839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.909864 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:46:12.909942 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.909943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.909950 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.909996 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.910223 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33658: BlockingRecv error: recv got EOF from 127.0.0.1:33658 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 04:46:12.910386 15284 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786337145367972-15284-0/krb5kdc/krb5cc' of type FILE
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.911298 15284 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:12.912336 15284 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:46:12 dist-test-slave-8hhm krb5kdc[24772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:46:12.918789 25765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:12.912855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.14.237.1:41503)
0810 04:46:12.912910 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:12.912912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:12.912971 (+ 59us) server_negotiation.cc:407] Connection header received
0810 04:46:12.913033 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:12.913034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:12.913041 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:12.913090 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:12.913492 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.913929 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.914559 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:12.914717 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:12.914770 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:12.916491 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:12.916492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:12.916492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:12.916498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:12.917837 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.918175 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.918176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.918176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.918223 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:12.918377 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:12.918378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:12.918378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:12.918454 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:12.918519 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:12.918598 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 04:46:12.918648 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:46:12.919428 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 24997
I20260810 04:46:12.932137 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 25254
I20260810 04:46:12.948647 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 25513
I20260810 04:46:12.961068 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 24800
2026-08-10T04:46:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2112 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:13 dist-test-slave-8hhm krb5kdc[25793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:13 dist-test-slave-8hhm krb5kdc[25793](info): set up 2 sockets
Aug 10 04:46:13 dist-test-slave-8hhm krb5kdc[25793](info): commencing operation
krb5kdc: starting...
W20260810 04:46:14.219755 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:14 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.62.keytab.
Entry for principal oryx/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.62.keytab.
I20260810 04:46:14.345052 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37911
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.62.keytab
--principal=oryx/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:37911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:14.442656 25818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:14.442847 25818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:14.442868 25818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:14.444021 25818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:14.444065 25818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:14.444080 25818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:14.444092 25818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:14.444145 25818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:14.445612 25818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:37911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.62.keytab
--principal=oryx/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:14.445855 25818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:14.446085 25818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:14.450781 25826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:14.450906 25818 server_base.cc:1034] running on GCE node
W20260810 04:46:14.450789 25825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:14.450997 25831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:14.451275 25818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:14.451565 25818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:14.452708 25818 hybrid_clock.cc:648] HybridClock initialized: now 1786337174452695 us; error 29 us; skew 500 ppm
Aug 10 04:46:14 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:14.455344 25818 init.cc:377] Logged in from keytab as oryx/127.14.237.62@KRBTEST.COM (short username oryx)
I20260810 04:46:14.455768 25818 webserver.cc:460] Webserver started at http://127.14.237.62:45365/ using document root <none> and password file <none>
I20260810 04:46:14.456005 25818 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:14.456048 25818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:14.456193 25818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:14.457281 25818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "1d10f2df2e8e4844bf70b1aca1360a61"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "c25f83a42080ecbcd761b3178f74a4e4"
server_key_iv: "8897f2a67eeef135539fdaeaf8a76edb"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.457662 25818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "1d10f2df2e8e4844bf70b1aca1360a61"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "c25f83a42080ecbcd761b3178f74a4e4"
server_key_iv: "8897f2a67eeef135539fdaeaf8a76edb"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.458958 25818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:14.459659 25846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:14.459853 25818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:14.459967 25818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "1d10f2df2e8e4844bf70b1aca1360a61"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "c25f83a42080ecbcd761b3178f74a4e4"
server_key_iv: "8897f2a67eeef135539fdaeaf8a76edb"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.460042 25818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:14.477821 25818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:14.480084 25818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:14.480245 25818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:14.484031 25818 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:37911
I20260810 04:46:14.484071 25951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:37911 every 8 connection(s)
I20260810 04:46:14.484436 25818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:14.484906 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 25818
I20260810 04:46:14.484995 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:14.485248 15284 external_mini_cluster.cc:1428] Setting key e875a98e0aaac696fd4b993da55e8ece
I20260810 04:46:14.485668 25952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:14.488770 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: Bootstrap starting.
I20260810 04:46:14.489491 25952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:14.489802 25952 log.cc:824] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:14.490551 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: No bootstrap required, opened a new log
Aug 10 04:46:14 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:14.492551 25952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } }
I20260810 04:46:14.492754 25952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:14.492798 25952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d10f2df2e8e4844bf70b1aca1360a61, State: Initialized, Role: FOLLOWER
I20260810 04:46:14.492941 25952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } }
I20260810 04:46:14.493031 25952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:14.493072 25952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:14.493132 25952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:14.493885 25952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } }
I20260810 04:46:14.493999 25952 leader_election.cc:302] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d10f2df2e8e4844bf70b1aca1360a61; no voters:
I20260810 04:46:14.494175 25952 leader_election.cc:288] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:14.494261 25965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:14.494427 25965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 1 LEADER]: Becoming Leader. State: Replica: 1d10f2df2e8e4844bf70b1aca1360a61, State: Running, Role: LEADER
I20260810 04:46:14.494457 25952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:14.494516 25965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } }
I20260810 04:46:14.494833 25966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } }
I20260810 04:46:14.494828 25965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d10f2df2e8e4844bf70b1aca1360a61. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } }
I20260810 04:46:14.494985 25966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:14.494998 25965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:14.495313 25979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:14.495821 25979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:14.496640 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:14.485753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.14.237.62:37911)
0810 04:46:14.485932 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:14.485941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:14.485972 (+ 31us) server_negotiation.cc:407] Connection header received
0810 04:46:14.486458 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:14.486467 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:14.486523 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:14.486666 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:14.487210 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.488469 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.489173 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.489812 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.489919 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:14.492214 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:14.492220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:14.492220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:14.492235 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:14.494396 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:14.494865 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:14.494868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:14.494869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:14.494987 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:14.495240 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:14.495240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:14.495241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:14.495563 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:14.496257 (+ 694us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:14.496314 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 04:46:14.496443 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260810 04:46:14.497521 25979 catalog_manager.cc:1347] Generated new cluster ID: 7b6c64b8ebc34fb79ce19293f2cd6cde
I20260810 04:46:14.497570 25979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:14.526952 25979 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:14.527668 25979 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:14.537561 25979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: Generated new TSK 0
I20260810 04:46:14.537781 25979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab.
Entry for principal oryx/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab.
I20260810 04:46:14.575755 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37911
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab
--principal=oryx/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:14.662056 25999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:14.662264 25999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:14.662287 25999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:14.663507 25999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:14.663560 25999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:14.663586 25999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:14.665000 25999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab
--principal=oryx/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:14.665196 25999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:14.665450 25999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:14.670503 26010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:14.670516 26012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:14.670511 26007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:14.670773 25999 server_base.cc:1034] running on GCE node
I20260810 04:46:14.670997 25999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:14.671303 25999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:14.672459 25999 hybrid_clock.cc:648] HybridClock initialized: now 1786337174672450 us; error 27 us; skew 500 ppm
Aug 10 04:46:14 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:14.675032 25999 init.cc:377] Logged in from keytab as oryx/127.14.237.1@KRBTEST.COM (short username oryx)
I20260810 04:46:14.675414 25999 webserver.cc:460] Webserver started at http://127.14.237.1:40015/ using document root <none> and password file <none>
I20260810 04:46:14.675608 25999 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:14.675647 25999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:14.675745 25999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:14.676757 25999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "6e895a0fbcfc488284197d37e3b9f250"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "a78e9e69a1f871eb362e85e4c8760cdb"
server_key_iv: "8abcacb8481139dfc4ba62330a7705ab"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.677045 25999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "6e895a0fbcfc488284197d37e3b9f250"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "a78e9e69a1f871eb362e85e4c8760cdb"
server_key_iv: "8abcacb8481139dfc4ba62330a7705ab"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.678511 25999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:14.679186 26023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:14.679360 25999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:14.679430 25999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "6e895a0fbcfc488284197d37e3b9f250"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "a78e9e69a1f871eb362e85e4c8760cdb"
server_key_iv: "8abcacb8481139dfc4ba62330a7705ab"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.679477 25999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:14.691596 25999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:14.693364 25999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:14.693475 25999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:14.693643 25999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:14.693956 25999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:14.693989 25999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:14.694011 25999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:14.694058 25999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:14.700152 25999 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:37009
I20260810 04:46:14.700227 26232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:37009 every 8 connection(s)
I20260810 04:46:14.700531 25999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:14.701761 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 25999
I20260810 04:46:14.701841 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:14.702049 15284 external_mini_cluster.cc:1428] Setting key 8da4b4438bd25bc11c04afcee25c26f1
Aug 10 04:46:14 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:14.710598 26239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:14.701350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.1:49321)
0810 04:46:14.701502 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:14.701538 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:14.702225 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:14.702476 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:14.702479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:14.703045 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:14.703251 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:14.703255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.703964 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.703965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:14.704772 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:14.704773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.705189 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.705190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:14.705355 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:14.705844 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:14.705854 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:14.707355 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:14.709295 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:14.709301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:14.709303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:14.709514 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:14.709742 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:14.709742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:14.709742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:14.709836 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:14.710128 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:14.710132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:14.710255 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 04:46:14.710333 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:46:14.710444 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260810 04:46:14.710597 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:14.701354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:49321 (local address 127.14.237.62:37911)
0810 04:46:14.701403 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:14.701405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:14.701653 (+ 248us) server_negotiation.cc:407] Connection header received
0810 04:46:14.702329 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:14.702330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:14.702343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:14.702404 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:14.703354 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.703863 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.704864 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.705085 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.705176 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:14.707449 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:14.707451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:14.707451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:14.707460 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:14.709176 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:14.709591 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:14.709592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:14.709593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:14.709664 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:14.709918 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:14.709919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:14.709920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:14.710063 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:14.710135 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:14.710382 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 04:46:14.710476 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:46:14.711205 26235 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37911
I20260810 04:46:14.711311 26235 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:14.711498 26235 heartbeater.cc:511] Master 127.14.237.62:37911 requested a full tablet report, sending...
I20260810 04:46:14.712091 25881 ts_manager.cc:194] Registered new tserver with Master: 6e895a0fbcfc488284197d37e3b9f250 (127.14.237.1:37009)
WARNING: no policy specified for oryx/127.14.237.2@KRBTEST.COM; defaulting to no policy
I20260810 04:46:14.712841 25881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:49321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab.
Entry for principal oryx/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab.
I20260810 04:46:14.728211 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37911
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab
--principal=oryx/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:14.808957 26245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:14.809156 26245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:14.809180 26245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:14.810297 26245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:14.810336 26245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:14.810361 26245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:14.811707 26245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab
--principal=oryx/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:14.811939 26245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:14.812129 26245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:14.817322 26260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:14.817349 26254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:14.817346 26252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:14.817521 26245 server_base.cc:1034] running on GCE node
I20260810 04:46:14.817744 26245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:14.818030 26245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:14.819192 26245 hybrid_clock.cc:648] HybridClock initialized: now 1786337174819181 us; error 31 us; skew 500 ppm
Aug 10 04:46:14 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:14.821862 26245 init.cc:377] Logged in from keytab as oryx/127.14.237.2@KRBTEST.COM (short username oryx)
I20260810 04:46:14.822270 26245 webserver.cc:460] Webserver started at http://127.14.237.2:43187/ using document root <none> and password file <none>
I20260810 04:46:14.822474 26245 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:14.822515 26245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:14.822603 26245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:14.823642 26245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "eb519794311a6da2b65916206bf8c8cd"
server_key_iv: "8abc0588258ef98b8bb2c97285058534"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.823968 26245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "eb519794311a6da2b65916206bf8c8cd"
server_key_iv: "8abc0588258ef98b8bb2c97285058534"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.825394 26245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:46:14.826117 26271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:14.826313 26245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:14.826380 26245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "eb519794311a6da2b65916206bf8c8cd"
server_key_iv: "8abc0588258ef98b8bb2c97285058534"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.826449 26245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:14.847745 26245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:14.849743 26245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:14.849856 26245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:14.850057 26245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:14.850380 26245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:14.850435 26245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:14.850463 26245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:14.850566 26245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:14.856547 26245 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:46599
I20260810 04:46:14.856601 26478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:46599 every 8 connection(s)
I20260810 04:46:14.856897 26245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:46:14 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:14.863073 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 26245
I20260810 04:46:14.863196 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:14.863463 15284 external_mini_cluster.cc:1428] Setting key c17bbdbe1b3047889c733c0a41d2e2e7
I20260810 04:46:14.866564 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:14.857597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:45233 (local address 127.14.237.62:37911)
0810 04:46:14.857670 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:14.857672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:14.857841 (+ 169us) server_negotiation.cc:407] Connection header received
0810 04:46:14.858329 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:14.858331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:14.858341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:14.858413 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:14.859304 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.859805 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.860745 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.860932 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.860985 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:14.863220 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:14.863222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:14.863223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:14.863232 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:14.864880 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:14.865440 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:14.865441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:14.865441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:14.865502 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:14.865799 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:14.865800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:14.865800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:14.865910 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:14.865966 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:14.866332 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 04:46:14.866430 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:46:14.866719 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:14.857532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.2:45233)
0810 04:46:14.857712 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:14.857754 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:14.858240 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:14.858486 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:14.858489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:14.859012 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:14.859208 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:14.859213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.859882 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.859882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:14.860666 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:14.860668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:14.861004 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:14.861005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:14.861099 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:14.861596 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:14.861604 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:14.863089 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:14.864975 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:14.864982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:14.864984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:14.865356 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:14.865585 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:14.865586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:14.865586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:14.865731 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:14.865976 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:14.865982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:14.866160 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 04:46:14.866288 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 04:46:14.866468 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260810 04:46:14.867450 26480 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37911
I20260810 04:46:14.867576 26480 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:14.867782 26480 heartbeater.cc:511] Master 127.14.237.62:37911 requested a full tablet report, sending...
I20260810 04:46:14.868283 25881 ts_manager.cc:194] Registered new tserver with Master: 5e2fc734fc3c4b2fb140b97a5a5ec651 (127.14.237.2:46599)
I20260810 04:46:14.868712 25881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:45233
WARNING: no policy specified for oryx/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab.
Entry for principal oryx/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab.
I20260810 04:46:14.891736 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37911
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab
--principal=oryx/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:14.973429 26487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:14.973635 26487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:14.973659 26487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:14.974737 26487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:14.974771 26487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:14.974797 26487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:14.976122 26487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab
--principal=oryx/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:37911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:14.976313 26487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:14.976536 26487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:14.981354 26502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:14.981328 26495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:14.981338 26499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:14.981765 26487 server_base.cc:1034] running on GCE node
I20260810 04:46:14.982012 26487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:14.982298 26487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:14.983474 26487 hybrid_clock.cc:648] HybridClock initialized: now 1786337174983468 us; error 35 us; skew 500 ppm
Aug 10 04:46:14 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:14.985826 26487 init.cc:377] Logged in from keytab as oryx/127.14.237.3@KRBTEST.COM (short username oryx)
I20260810 04:46:14.986249 26487 webserver.cc:460] Webserver started at http://127.14.237.3:38169/ using document root <none> and password file <none>
I20260810 04:46:14.986471 26487 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:14.986536 26487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:14.986685 26487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:14.987643 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "cc7e65bae65841319631e23b85e7fdf8"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "f017f58c403e1776590a4d25e0d54943"
server_key_iv: "a93e3e7e73f9cf7beacf9240929e0806"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.987982 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "cc7e65bae65841319631e23b85e7fdf8"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "f017f58c403e1776590a4d25e0d54943"
server_key_iv: "a93e3e7e73f9cf7beacf9240929e0806"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.989336 26487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:46:14.990043 26514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:14.990235 26487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:14.990334 26487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "cc7e65bae65841319631e23b85e7fdf8"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "f017f58c403e1776590a4d25e0d54943"
server_key_iv: "a93e3e7e73f9cf7beacf9240929e0806"
server_key_version: "encryptionkey@0"
I20260810 04:46:14.990429 26487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.007566 26487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:15.009430 26487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:15.009588 26487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:15.009845 26487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:15.010159 26487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:15.010228 26487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.010273 26487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:15.010306 26487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.016839 26487 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:39199
I20260810 04:46:15.016875 26748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:39199 every 8 connection(s)
I20260810 04:46:15.017271 26487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:15.026180 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.018062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:48327 (local address 127.14.237.62:37911)
0810 04:46:15.018132 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.018134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.018312 (+ 178us) server_negotiation.cc:407] Connection header received
0810 04:46:15.018787 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.018789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.018799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.018855 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.019771 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.020252 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.021171 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.021432 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.021483 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.023634 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.023636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.023636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.023643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.025001 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.025447 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.025448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.025448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.025503 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.025719 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.025720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.025720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.025807 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.025851 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.026055 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.026116 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:46:15.026432 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.018038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.3:48327)
0810 04:46:15.018188 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.018229 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.018705 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.018915 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.018918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.019503 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.019693 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.019698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.020334 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.020335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.021087 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.021089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.021501 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.021502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.021606 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.022112 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.022120 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.023547 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.025072 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.025078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.025080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.025391 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.025560 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.025560 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.025561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.025673 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.025854 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.025858 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.025962 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.026043 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.026181 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260810 04:46:15.027112 26749 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37911
I20260810 04:46:15.027233 26749 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:15.027273 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 26487
I20260810 04:46:15.027365 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:15.027452 26749 heartbeater.cc:511] Master 127.14.237.62:37911 requested a full tablet report, sending...
I20260810 04:46:15.027577 15284 external_mini_cluster.cc:1428] Setting key da3ddfa66a143d5c7320670fcaff6369
I20260810 04:46:15.027884 25881 ts_manager.cc:194] Registered new tserver with Master: cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199)
I20260810 04:46:15.028345 25881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:48327
I20260810 04:46:15.028491 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.237.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.61.keytab.
Entry for principal oryx/127.14.237.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.61.keytab.
I20260810 04:46:15.055639 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.61:41095
--webserver_interface=127.14.237.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.61.keytab
--principal=oryx/127.14.237.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:37911,127.14.237.61:41095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:15.137444 26761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:15.137650 26761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:15.137672 26761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:15.138697 26761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:15.138728 26761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:15.138741 26761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:15.138754 26761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:15.138765 26761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:15.140115 26761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:37911,127.14.237.61:41095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.61.keytab
--principal=oryx/127.14.237.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.61:41095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.237.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
W20260810 04:46:15.140214 26761 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.237.62:37911,127.14.237.61:41095'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 04:46:15.140318 26761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:15.140556 26761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:15.145503 26776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.145512 26769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.145516 26773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:15.145576 26761 server_base.cc:1034] running on GCE node
I20260810 04:46:15.145959 26761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:15.146210 26761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:15.147373 26761 hybrid_clock.cc:648] HybridClock initialized: now 1786337175147355 us; error 28 us; skew 500 ppm
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:15.149874 26761 init.cc:377] Logged in from keytab as oryx/127.14.237.61@KRBTEST.COM (short username oryx)
I20260810 04:46:15.150357 26761 webserver.cc:460] Webserver started at http://127.14.237.61:46795/ using document root <none> and password file <none>
I20260810 04:46:15.150616 26761 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.150687 26761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.150832 26761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:15.151928 26761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data/instance:
uuid: "02f9304665a3462c8bdb206d08d2a0b1"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "64f8b74817eb817d682095fab29adcc8"
server_key_iv: "fe2a33ef0548e920805e914bb7be9a17"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.152298 26761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal/instance:
uuid: "02f9304665a3462c8bdb206d08d2a0b1"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "64f8b74817eb817d682095fab29adcc8"
server_key_iv: "fe2a33ef0548e920805e914bb7be9a17"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.153726 26761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:15.154434 26787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.154599 26761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:15.154707 26761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
uuid: "02f9304665a3462c8bdb206d08d2a0b1"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "64f8b74817eb817d682095fab29adcc8"
server_key_iv: "fe2a33ef0548e920805e914bb7be9a17"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.154780 26761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.165155 26761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:15.166923 26761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:15.167078 26761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:15.170985 26761 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.61:41095
I20260810 04:46:15.171046 26878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.61:41095 every 8 connection(s)
I20260810 04:46:15.171366 26761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
I20260810 04:46:15.172348 26879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:15.173951 26879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.61@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:15.181105 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 26761
I20260810 04:46:15.182518 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.174668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.14.237.62:37911)
0810 04:46:15.174734 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.174735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.174915 (+ 180us) server_negotiation.cc:407] Connection header received
0810 04:46:15.175314 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.175315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.175324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.175380 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.175913 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.176385 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.177346 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.177524 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.177577 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.179646 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.179648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.179648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.179655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.180839 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.181332 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.181334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.181334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.181398 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.181727 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.181728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.181729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.181966 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.182045 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.182391 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.182458 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:46:15.182761 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.174623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:33676)
0810 04:46:15.174796 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.174834 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.175239 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.175449 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.175452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.175679 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.175848 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.175852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.176457 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.176458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.177268 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.177271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.177593 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.177594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.177702 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.178216 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.178224 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.179562 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.180908 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.180914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.180916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.181138 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.181468 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.181469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.181469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.181595 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.182044 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.182049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.182237 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.182371 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.182533 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260810 04:46:15.183389 26879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.237.61@KRBTEST.COM
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.61@KRBTEST.COM for oryx/127.14.237.61@KRBTEST.COM
I20260810 04:46:15.190757 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.181789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.14.237.61:41095)
0810 04:46:15.181994 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.181998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.182021 (+ 23us) server_negotiation.cc:407] Connection header received
0810 04:46:15.182067 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.182069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.182081 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.182148 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.182890 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.183952 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.184668 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.184976 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.185051 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.187410 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.187418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.187418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.187424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.189440 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.189793 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.189796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.189796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.189863 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.190066 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.190067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.190067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.190255 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.190421 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.190528 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.190626 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":108,"threads_started":1}
I20260810 04:46:15.191363 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.183609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.0.0.1:44352)
0810 04:46:15.183689 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.183699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.183757 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.183961 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.183961 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.184085 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.184126 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.184128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.184765 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.184766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.185317 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.185320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.185699 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.185700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.185764 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.186151 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.186155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.187735 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.190144 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.190145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.190145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.190355 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.190570 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.190571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.190571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.190638 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.191035 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.191035 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.191085 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.191158 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.191254 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 04:46:15.191465 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.183690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.14.237.61:41095)
0810 04:46:15.183843 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.183845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.183853 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:46:15.183870 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.183871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.183880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.183944 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.184203 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.184669 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.185422 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.185632 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.185700 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.187882 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.187884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.187884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.187890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.190086 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.190435 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.190436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.190437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.190503 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.190795 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.190796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.190796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.190954 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.191014 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.191196 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.191309 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260810 04:46:15.192654 26879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Bootstrap starting.
I20260810 04:46:15.193336 26879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:15.193585 26879 log.cc:824] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:15.194332 26879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: No bootstrap required, opened a new log
I20260810 04:46:15.195909 26879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } }
I20260810 04:46:15.196025 26879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:15.196046 26879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f9304665a3462c8bdb206d08d2a0b1, State: Initialized, Role: FOLLOWER
I20260810 04:46:15.196170 26879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } }
I20260810 04:46:15.196393 26893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } } }
I20260810 04:46:15.196460 26879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:15.196480 26893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:46:15.198292 26912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:46:15.198369 26912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:46:15.199747 26761 master_runner.cc:428] Detected that this master 02f9304665a3462c8bdb206d08d2a0b1 is joining an existing cluster
I20260810 04:46:15.199851 26761 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.237.61:41095
I20260810 04:46:15.206957 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.200841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.14.237.62:37911)
0810 04:46:15.200890 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.200893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.201020 (+ 127us) server_negotiation.cc:407] Connection header received
0810 04:46:15.201091 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.201092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.201103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.201151 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.201604 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.202102 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.202776 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.202963 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.203034 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.204073 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.204075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.204075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.204083 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.205712 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.206131 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.206132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.206132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.206194 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.206397 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.206398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.206398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.206528 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.206597 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.206759 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.206840 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:46:15.207074 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.200783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:33690)
0810 04:46:15.200951 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.200959 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.201031 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.201273 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.201274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.201477 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.201525 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.201526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.202177 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.202177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.202691 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.202693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.203045 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.203046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.203104 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.203588 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.203593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.203970 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.205803 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.205804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.205804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.206021 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.206271 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.206272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.206272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.206332 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.206585 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.206586 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.206630 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.206711 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.206808 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":130,"threads_started":1}
I20260810 04:46:15.207696 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.200856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.14.237.61:41095)
0810 04:46:15.200897 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.200899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.201031 (+ 132us) server_negotiation.cc:407] Connection header received
0810 04:46:15.201120 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.201121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.201129 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.201175 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.201601 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.202321 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.203288 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.203574 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.203611 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.204603 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.204604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.204604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.204611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.206469 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.206852 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.206853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.206853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.206902 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.207081 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.207082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.207083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.207243 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.207310 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.207548 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.207620 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:46:15.207963 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.200797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.0.0.1:44368)
0810 04:46:15.200974 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.200984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.201053 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.201183 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.201183 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.201451 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.201519 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.201521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.202403 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.202403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.203180 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.203182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.203677 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.203678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.203732 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.204140 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.204145 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.204511 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.206559 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.206560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.206560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.206772 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.206961 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.206961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.206961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.207011 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.207401 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.207402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.207445 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.207797 (+ 352us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.207878 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.62@KRBTEST.COM for oryx/127.14.237.61@KRBTEST.COM
I20260810 04:46:15.216609 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.209699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.14.237.61:41095)
0810 04:46:15.209737 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.209739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.209895 (+ 156us) server_negotiation.cc:407] Connection header received
0810 04:46:15.209949 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.209950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.209960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.210022 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.210396 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.211063 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.211811 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.212129 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.212199 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.214115 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.214116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.214117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.214124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.215395 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.215778 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.215779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.215780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.215853 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.216070 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.216070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.216071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.216175 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.216244 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.216451 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.216522 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:46:15.216631 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.209630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.0.0.1:44382)
0810 04:46:15.209764 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.209806 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.209893 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.210025 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.210026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.210281 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.210326 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.210327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.211175 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.211176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.211722 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.211724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.212207 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.212208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.212272 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.212691 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.212695 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.214016 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.215482 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.215490 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.215492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.215696 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.215931 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.215931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.215932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.215999 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.216260 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.216267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.216333 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.216412 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.216526 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":100,"threads_started":1}
I20260810 04:46:15.217636 25881 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.237.61:41095: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1"
member_type: NON_VOTER
last_known_addr {
host: "127.14.237.61"
port: 41095
}
attrs {
promote: true
}
}
dest_uuid: "1d10f2df2e8e4844bf70b1aca1360a61"
cas_config_opid_index: -1
I20260810 04:46:15.217978 25881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } }
I20260810 04:46:15.218442 26931 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 02f9304665a3462c8bdb206d08d2a0b1 (127.14.237.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } }
I20260810 04:46:15.218763 26837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:15.219385 26931 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.237.61:41095
I20260810 04:46:15.219471 26931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } }
I20260810 04:46:15.219535 26931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:15.219671 26837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Refusing update from remote peer 1d10f2df2e8e4844bf70b1aca1360a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 04:46:15.219975 26931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:15.220129 26761 master.cc:537] Master@127.14.237.61:41095 shutting down...
W20260810 04:46:15.220847 25848 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 04:46:15.221346 25848 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 -> Peer 02f9304665a3462c8bdb206d08d2a0b1 (127.14.237.61:41095): Couldn't send request to peer 02f9304665a3462c8bdb206d08d2a0b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 04:46:15.222795 26761 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:15.222900 26761 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:15.222952 26761 tablet_replica.cc:330] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: stopping tablet replica
I20260810 04:46:15.235479 26761 master.cc:552] Master@127.14.237.61:41095 shutdown complete.
I20260810 04:46:15.235526 26761 master_runner.cc:305] Clearing existing system tablet
I20260810 04:46:15.235670 26761 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.235711 26761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.236416 26761 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260810 04:46:15.236902 26938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.237023 26761 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.237082 26761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.237103 26761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
uuid: "02f9304665a3462c8bdb206d08d2a0b1"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "64f8b74817eb817d682095fab29adcc8"
server_key_iv: "fe2a33ef0548e920805e914bb7be9a17"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.237814 26761 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:15.239346 26761 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 04:46:15.239406 26761 log.cc:1196] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 04:46:15.239611 26761 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Deleting consensus metadata
I20260810 04:46:15.239749 26761 master_runner.cc:315] Copying system tablet from 127.14.237.62:37911
I20260810 04:46:15.240419 26761 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: tablet copy: Beginning tablet copy session from remote peer at address 127.14.237.62:37911
I20260810 04:46:15.246755 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.240847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.14.237.62:37911)
0810 04:46:15.240896 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.240898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.241015 (+ 117us) server_negotiation.cc:407] Connection header received
0810 04:46:15.241050 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.241052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.241063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.241149 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.241759 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.242293 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.242986 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.243191 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.243293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.244256 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.244257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.244258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.244265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.245651 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.245992 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.245993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.245994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.246054 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.246256 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.246257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.246257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.246366 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.246438 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.246601 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.246670 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:46:15.246779 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.240775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:33694)
0810 04:46:15.240938 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.240953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.241008 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.241151 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.241152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.241436 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.241492 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.241494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.242379 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.242380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.242901 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.242903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.243291 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.243292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.243363 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.243872 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.243877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.244163 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.245740 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.245741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.245742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.245912 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.246121 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.246122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.246123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.246187 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.246440 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.246441 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.246485 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.246561 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.246667 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260810 04:46:15.247253 25921 tablet_copy_service.cc:140] P 1d10f2df2e8e4844bf70b1aca1360a61: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 02f9304665a3462c8bdb206d08d2a0b1 ({username='oryx', principal='oryx/127.14.237.61@KRBTEST.COM'} at 127.0.0.1:33694)
I20260810 04:46:15.247334 25921 tablet_copy_service.cc:161] P 1d10f2df2e8e4844bf70b1aca1360a61: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 02f9304665a3462c8bdb206d08d2a0b1 at {username='oryx', principal='oryx/127.14.237.61@KRBTEST.COM'} at 127.0.0.1:33694: session id = 02f9304665a3462c8bdb206d08d2a0b1-00000000000000000000000000000000
I20260810 04:46:15.247951 25921 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 04:46:15.248570 26761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 04:46:15.249563 26761 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: tablet copy: Starting download of 0 data blocks...
I20260810 04:46:15.249711 26761 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: tablet copy: Starting download of 1 WAL segments...
I20260810 04:46:15.250560 26761 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 04:46:15.251336 25921 tablet_copy_service.cc:342] P 1d10f2df2e8e4844bf70b1aca1360a61: Request end of tablet copy session 02f9304665a3462c8bdb206d08d2a0b1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.237.61@KRBTEST.COM'} at 127.0.0.1:33694
I20260810 04:46:15.251398 25921 tablet_copy_service.cc:433] P 1d10f2df2e8e4844bf70b1aca1360a61: ending tablet copy session 02f9304665a3462c8bdb206d08d2a0b1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 02f9304665a3462c8bdb206d08d2a0b1
I20260810 04:46:15.253800 26761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:15.254930 26955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.254910 26951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.255055 26957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:15.255255 26761 server_base.cc:1034] running on GCE node
I20260810 04:46:15.255342 26761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:15.255470 26761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:15.256595 26761 hybrid_clock.cc:648] HybridClock initialized: now 1786337175256590 us; error 33 us; skew 500 ppm
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:15.257903 26761 init.cc:377] Logged in from keytab as oryx/127.14.237.61@KRBTEST.COM (short username oryx)
I20260810 04:46:15.258190 26761 webserver.cc:460] Webserver started at http://127.14.237.61:38989/ using document root <none> and password file <none>
I20260810 04:46:15.258363 26761 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.258416 26761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.259012 26761 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:15.259545 26968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.259668 26761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:15.259729 26761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
uuid: "02f9304665a3462c8bdb206d08d2a0b1"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "64f8b74817eb817d682095fab29adcc8"
server_key_iv: "fe2a33ef0548e920805e914bb7be9a17"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.259785 26761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.268558 26761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:15.270198 26761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:15.273346 26761 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.61:41095
I20260810 04:46:15.273689 27057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.61:41095 every 8 connection(s)
I20260810 04:46:15.273774 26761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
I20260810 04:46:15.274662 27058 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:46:15.275229 27058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Bootstrap starting.
I20260810 04:46:15.278412 27058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:46:15.278693 27058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Bootstrap complete.
I20260810 04:46:15.278928 27058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } }
I20260810 04:46:15.279026 27058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 02f9304665a3462c8bdb206d08d2a0b1, State: Initialized, Role: LEARNER
I20260810 04:46:15.279066 27058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } }
I20260810 04:46:15.279271 27058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:15.279264 27062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } }
I20260810 04:46:15.279379 27062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: This master's current role is: LEARNER
I20260810 04:46:15.280959 27084 catalog_manager.cc:1259] Loaded cluster ID: 7b6c64b8ebc34fb79ce19293f2cd6cde
I20260810 04:46:15.281042 27084 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: loading cluster ID for follower catalog manager: success
I20260810 04:46:15.282035 27084 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: acquiring CA information for follower catalog manager: success
I20260810 04:46:15.282311 27084 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:46:15.300002 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.292398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.14.237.61:41095)
0810 04:46:15.292514 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.292516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.292530 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:46:15.292555 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.292555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.292564 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.292633 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.293453 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.294169 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.295401 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.295600 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.295649 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.297297 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.297298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.297298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.297304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.298745 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.299199 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.299200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.299200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.299265 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.299494 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.299494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.299495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.299593 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.299668 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.299847 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.299912 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.14.237.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.60.keytab.
Entry for principal oryx/127.14.237.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.60.keytab.
I20260810 04:46:15.332113 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.60:37137
--webserver_interface=127.14.237.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.60.keytab
--principal=oryx/127.14.237.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:37911,127.14.237.61:41095,127.14.237.60:37137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:15.428758 27090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:15.429055 27090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:15.429145 27090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:15.430475 27090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:15.430518 27090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:15.430532 27090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:15.430545 27090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:15.430557 27090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:15.432296 27090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:37911,127.14.237.61:41095,127.14.237.60:37137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.60.keytab
--principal=oryx/127.14.237.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.60:37137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.237.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:15.432509 27090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:15.432694 27090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:15.437677 27106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.437747 27104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.437697 27110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:15.437742 27090 server_base.cc:1034] running on GCE node
I20260810 04:46:15.438040 27090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:15.438407 27090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:15.439570 27090 hybrid_clock.cc:648] HybridClock initialized: now 1786337175439558 us; error 28 us; skew 500 ppm
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:15.442248 27090 init.cc:377] Logged in from keytab as oryx/127.14.237.60@KRBTEST.COM (short username oryx)
I20260810 04:46:15.442768 27090 webserver.cc:460] Webserver started at http://127.14.237.60:38921/ using document root <none> and password file <none>
I20260810 04:46:15.443004 27090 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.443048 27090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.443230 27090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:15.444296 27090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data/instance:
uuid: "191bdb4fb70340f08dd2629ee4138671"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "96f0c16b91da7bf111ea5be88cab9bb2"
server_key_iv: "55061c5d47415adf89f505b482484d56"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.444675 27090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal/instance:
uuid: "191bdb4fb70340f08dd2629ee4138671"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "96f0c16b91da7bf111ea5be88cab9bb2"
server_key_iv: "55061c5d47415adf89f505b482484d56"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.446058 27090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:15.446786 27119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.446971 27090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.447063 27090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
uuid: "191bdb4fb70340f08dd2629ee4138671"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "96f0c16b91da7bf111ea5be88cab9bb2"
server_key_iv: "55061c5d47415adf89f505b482484d56"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.447153 27090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.454607 27090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:15.456323 27090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:15.456478 27090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:15.460798 27090 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.60:37137
I20260810 04:46:15.460871 27219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.60:37137 every 8 connection(s)
I20260810 04:46:15.461294 27090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
I20260810 04:46:15.462440 27220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:15.464178 27220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:46:15.468600 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 27090
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.60@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:15.473479 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.465118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.14.237.62:37911)
0810 04:46:15.465172 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.465173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.465251 (+ 78us) server_negotiation.cc:407] Connection header received
0810 04:46:15.465727 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.465728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.465738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.465797 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.466406 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.466873 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.467780 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.467959 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.468012 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.470612 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.470614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.470614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.470622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.472024 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.472439 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.472441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.472441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.472502 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.472772 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.472772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.472773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.472885 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.472955 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.473285 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.473376 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:46:15.473714 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.464876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:33710)
0810 04:46:15.465094 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.465135 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.465632 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.465907 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.465911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.466151 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.466330 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.466333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.466948 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.466949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.467707 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.467709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.468042 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.468043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.468152 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.468669 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.468681 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.470519 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.472120 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.472128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.472130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.472358 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.472573 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.472573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.472574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.472690 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.472959 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.472966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.473096 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.473322 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.473475 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":121,"threads_started":1}
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.237.60@KRBTEST.COM
I20260810 04:46:15.474263 27220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:46:15.478183 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.469186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.14.237.60:37137)
0810 04:46:15.469373 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.469378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.469401 (+ 23us) server_negotiation.cc:407] Connection header received
0810 04:46:15.469426 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.469429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.469448 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.469519 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.470198 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.470860 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.471756 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.472078 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.472158 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.474709 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.474717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.474717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.474724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.476854 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.477287 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.477290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.477291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.477355 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.477571 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.477571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.477571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.477711 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.477849 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.477899 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.477989 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":125,"threads_started":1}
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.60@KRBTEST.COM for oryx/127.14.237.61@KRBTEST.COM
I20260810 04:46:15.481091 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.474535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.0.0.1:44422)
0810 04:46:15.474576 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.474586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.474629 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.474785 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.474787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.474932 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.475004 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.475006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.475755 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.475755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.476375 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.476377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.476763 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.476764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.476833 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.477179 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.477184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.478566 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.480195 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.480196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.480196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.480418 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.480567 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.480568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.480568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.480615 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.480833 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.480833 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.480887 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.480945 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.481024 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 04:46:15.481194 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.474588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.14.237.61:41095)
0810 04:46:15.474629 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.474631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.474639 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:46:15.474688 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.474689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.474700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.474766 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.475090 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.475667 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.476477 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.476680 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.476761 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.478649 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.478650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.478650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.478657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.480133 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.480464 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.480465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.480465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.480513 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.480650 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.480650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.480651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.480751 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.480813 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.480987 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.481080 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:46:15.481554 27220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.60@KRBTEST.COM for oryx/127.14.237.60@KRBTEST.COM
I20260810 04:46:15.487783 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.481790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:37137 (local address 127.0.0.1:40930)
0810 04:46:15.481839 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.481849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.481890 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.481991 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.481992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.482093 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.482124 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.482125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.482775 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.482775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.483299 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.483301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.483641 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.483642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.483686 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.484147 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.484152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.485463 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.486970 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.486971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.486971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.487133 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.487310 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.487311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.487311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.487367 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.487577 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.487578 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.487619 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.487670 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.487730 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 04:46:15.487845 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.481837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.14.237.60:37137)
0810 04:46:15.481900 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.481901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.481910 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:15.481936 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.481936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.481944 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.481985 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.482183 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.482712 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.483367 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.483544 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.483594 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.485550 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.485551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.485551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.485558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.486893 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.487198 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.487198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.487199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.487255 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.487441 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.487442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.487442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.487526 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.487572 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.487710 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.487773 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":46}
I20260810 04:46:15.489030 27220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Bootstrap starting.
I20260810 04:46:15.489913 27220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:15.490204 27220 log.cc:824] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:15.490926 27220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: No bootstrap required, opened a new log
I20260810 04:46:15.492395 27220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } }
I20260810 04:46:15.492515 27220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:15.492537 27220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191bdb4fb70340f08dd2629ee4138671, State: Initialized, Role: FOLLOWER
I20260810 04:46:15.492684 27220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } }
I20260810 04:46:15.493059 27233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } } }
I20260810 04:46:15.493114 27220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:15.493166 27233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:46:15.495285 27250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:46:15.495375 27250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:46:15.497542 27090 master_runner.cc:428] Detected that this master 191bdb4fb70340f08dd2629ee4138671 is joining an existing cluster
I20260810 04:46:15.497594 27090 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.237.60:37137
I20260810 04:46:15.506114 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.498633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.14.237.62:37911)
0810 04:46:15.498694 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.498696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.499041 (+ 345us) server_negotiation.cc:407] Connection header received
0810 04:46:15.499324 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.499325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.499335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.499365 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.499815 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.500599 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.501729 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.502018 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.502062 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.503446 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.503447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.503447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.503454 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.504866 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.505344 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.505345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.505346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.505417 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.505641 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.505642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.505643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.505750 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.505805 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.505987 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.506050 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:46:15.506181 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.498610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:33722)
0810 04:46:15.498976 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.498986 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.499242 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.499425 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.499426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.499649 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.499720 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.499722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.500705 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.500706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.501607 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.501609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.502478 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.502479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.502547 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.502903 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.502910 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.503354 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.504952 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.504953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.504953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.505194 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.505497 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.505498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.505498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.505571 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.505822 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.505823 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.505880 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.505956 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.506058 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":179,"threads_started":1}
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337174, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.62@KRBTEST.COM for oryx/127.14.237.60@KRBTEST.COM
I20260810 04:46:15.525516 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.508078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:37137 (local address 127.0.0.1:40946)
0810 04:46:15.508131 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.508141 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.508182 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.508454 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.508455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.508597 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.508630 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.508631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.509317 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.509318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.509836 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.509838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.510242 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.510244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.510311 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.510889 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.510894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.512336 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.524231 (+ 11895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.524234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.524492 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.524743 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.524744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.524812 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.525125 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.525126 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.525180 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.525257 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.525395 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:46:15.525529 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.499014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.14.237.61:41095)
0810 04:46:15.499085 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.499088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.499108 (+ 20us) server_negotiation.cc:407] Connection header received
0810 04:46:15.499131 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.499132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.499143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.499200 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.499532 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.500344 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.501437 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.501755 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.501793 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.522505 (+ 20712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.522507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.522508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.522518 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.524272 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524663 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.524664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.524665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.524719 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524928 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.524929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.524929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.525057 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.525103 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.525364 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.525441 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:46:15.525954 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.498861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:37137 (local address 127.0.0.1:40938)
0810 04:46:15.499031 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.499040 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.499089 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.499548 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.499549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.499765 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.499826 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.499828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.500653 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.500653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.501495 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.501496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.501947 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.501947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.501999 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.502439 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.502443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.522120 (+ 19677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.524938 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.524940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.525144 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.525350 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.525351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.525351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.525401 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.525645 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.525645 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.525675 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.525728 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.525840 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260810 04:46:15.526059 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.498602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.0.0.1:44438)
0810 04:46:15.498764 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.498775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.498823 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.499202 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.499202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.499394 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.499449 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.499450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.500430 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.500431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.501341 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.501343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.501885 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.501885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.501946 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.521774 (+ 19828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.521781 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.522347 (+ 566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.524360 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.524361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.524578 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.524794 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.524794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.524853 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.525151 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.525152 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.525202 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.525735 (+ 533us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.525930 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":54,"thread_start_us":109,"threads_started":1}
I20260810 04:46:15.526216 27228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.498895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.14.237.60:37137)
0810 04:46:15.498934 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.498935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.499079 (+ 144us) server_negotiation.cc:407] Connection header received
0810 04:46:15.499142 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.499143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.499151 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.499200 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.499917 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.500573 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.501562 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.501849 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.501887 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.522481 (+ 20594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.522483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.522483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.522490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.524846 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.525237 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.525238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.525238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.525288 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.525458 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.525458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.525459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.525575 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.525626 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.526069 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.526144 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:46:15.527019 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.508152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.14.237.60:37137)
0810 04:46:15.508331 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.508333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.508339 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:46:15.508365 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.508366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.508373 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.508438 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.508702 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.509206 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.509937 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.510161 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.510239 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.512441 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.512442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.512443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.512449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.524102 (+ 11653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524584 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.524585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.524586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.524648 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.524888 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.524888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.524889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.525039 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.526804 (+ 1765us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.526846 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.526923 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260810 04:46:15.527478 25881 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.237.60:37137: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "191bdb4fb70340f08dd2629ee4138671"
member_type: NON_VOTER
last_known_addr {
host: "127.14.237.60"
port: 37137
}
attrs {
promote: true
}
}
dest_uuid: "1d10f2df2e8e4844bf70b1aca1360a61"
cas_config_opid_index: 5
I20260810 04:46:15.527734 25881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } }
I20260810 04:46:15.528113 26932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } } }
I20260810 04:46:15.528375 27176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:15.529232 26932 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.237.60:37137
I20260810 04:46:15.529323 27176 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Refusing update from remote peer 1d10f2df2e8e4844bf70b1aca1360a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 04:46:15.529313 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } } }
I20260810 04:46:15.529376 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:15.529577 26932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:15.529991 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 25999
I20260810 04:46:15.532008 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.528193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.0.0.1:44446)
0810 04:46:15.528308 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.528318 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.528353 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.528566 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.528567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.528569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:15.528613 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.528615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.529516 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.529517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.530829 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.530832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.531728 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.531729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.531808 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.531828 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.531900 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.531937 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 04:46:15.532292 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.528275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.14.237.61:41095)
0810 04:46:15.528415 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.528417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.528423 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:46:15.528444 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.528444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.528465 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.528517 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:15.528680 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.529424 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.530927 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.531557 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.531624 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.531693 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.532170 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.532217 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 04:46:15.533283 27176 raft_consensus.cc:858] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } }
W20260810 04:46:15.533394 27176 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } }
I20260810 04:46:15.537442 27266 mvcc.cc:205] Tried to move back new op lower bound from 7316837066738192384 to 7316837066729947136. Current Snapshot: MvccSnapshot[applied={T|T < 7316837066738192384}]
I20260810 04:46:15.539369 27176 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 02f9304665a3462c8bdb206d08d2a0b1 (127.14.237.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } }
I20260810 04:46:15.542017 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } } }
I20260810 04:46:15.542122 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:15.542466 27175 raft_consensus.cc:1467] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260810 04:46:15.542564 27175 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } } }
I20260810 04:46:15.544435 26931 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: attempting to promote NON_VOTER 191bdb4fb70340f08dd2629ee4138671 to VOTER
I20260810 04:46:15.544718 26931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:15.545651 27010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Refusing update from remote peer 1d10f2df2e8e4844bf70b1aca1360a61: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 04:46:15.545979 26932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 04:46:15.546622 27010 raft_consensus.cc:858] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
W20260810 04:46:15.546705 27010 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:15.547289 27010 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } } }
I20260810 04:46:15.549268 27062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d10f2df2e8e4844bf70b1aca1360a61. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } } }
I20260810 04:46:15.549387 27062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: This master's current role is: LEARNER
I20260810 04:46:15.549438 27062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: NON_VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: true } } }
I20260810 04:46:15.549517 27062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: This master's current role is: LEARNER
I20260810 04:46:15.549659 27175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 LEARNER]: Refusing update from remote peer 1d10f2df2e8e4844bf70b1aca1360a61: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 04:46:15.549927 26932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 04:46:15.550716 26932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.551999 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.552105 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:15.552174 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.552264 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:15.552651 27175 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.554595 27276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.554729 27276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:15.554783 27276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.554868 27276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:15.554920 27276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.554998 27276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:15.558362 27233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.558445 27233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:15.558550 27269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d10f2df2e8e4844bf70b1aca1360a61. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.558602 27269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:15.560482 27090 master.cc:537] Master@127.14.237.60:37137 shutting down...
I20260810 04:46:15.561635 27278 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: attempting to promote NON_VOTER 02f9304665a3462c8bdb206d08d2a0b1 to VOTER
I20260810 04:46:15.561775 27278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:15.562074 27278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } } pending_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.562180 27278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:15.562765 27010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Refusing update from remote peer 1d10f2df2e8e4844bf70b1aca1360a61: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 04:46:15.563016 26932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 04:46:15.562943 27011 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.564214 27283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: NON_VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: true } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.564316 27283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: This master's current role is: LEARNER
W20260810 04:46:15.565380 25848 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 -> Peer 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60:37137): Couldn't send request to peer 191bdb4fb70340f08dd2629ee4138671. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 04:46:15.565659 27010 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.8] Dedup: 1.7->[1.8-1.8]
I20260810 04:46:15.566196 26932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02f9304665a3462c8bdb206d08d2a0b1 (127.14.237.61) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.566293 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:37009
--local_ip_for_outbound_sockets=127.14.237.1
--tserver_master_addrs=127.14.237.62:37911
--webserver_port=40015
--webserver_interface=127.14.237.1
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab
--principal=oryx/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095,127.14.237.60:37137 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:46:15.566339 27010 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02f9304665a3462c8bdb206d08d2a0b1 (127.14.237.61) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.567083 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.567165 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:15.567215 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.567255 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:15.567377 27283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.567467 27283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:15.568439 27090 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:15.568584 27090 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:15.568629 27090 tablet_replica.cc:330] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: stopping tablet replica
I20260810 04:46:15.582266 27090 master.cc:552] Master@127.14.237.60:37137 shutdown complete.
I20260810 04:46:15.582329 27090 master_runner.cc:305] Clearing existing system tablet
I20260810 04:46:15.582461 27090 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.582511 27090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.583364 27090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:46:15.583971 27294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.584146 27090 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.584245 27090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:15.584278 27090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
uuid: "191bdb4fb70340f08dd2629ee4138671"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "96f0c16b91da7bf111ea5be88cab9bb2"
server_key_iv: "55061c5d47415adf89f505b482484d56"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.585005 27090 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:15.586541 27090 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 04:46:15.586614 27090 log.cc:1196] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 04:46:15.586866 27090 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Deleting consensus metadata
I20260810 04:46:15.586987 27090 master_runner.cc:315] Copying system tablet from 127.14.237.62:37911
I20260810 04:46:15.587603 27090 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: tablet copy: Beginning tablet copy session from remote peer at address 127.14.237.62:37911
I20260810 04:46:15.596602 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.587904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:33726)
0810 04:46:15.589314 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.589327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.589409 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.589569 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.589570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.589823 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.589878 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.589880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.590841 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.590842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.591647 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.591649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.592158 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.592159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.592224 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.592846 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.592853 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.593277 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.595358 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.595360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.595360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.595584 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.595851 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.595852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.595853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.595924 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.596276 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.596277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.596328 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.596402 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.596505 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":75,"threads_started":1}
I20260810 04:46:15.596629 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.587979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.14.237.62:37911)
0810 04:46:15.588033 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.588036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.589418 (+ 1382us) server_negotiation.cc:407] Connection header received
0810 04:46:15.589463 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.589464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.589479 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.589555 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.589978 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.590734 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.591743 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.592056 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.592151 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.593406 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.593408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.593409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.593417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.595254 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.595702 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.595704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.595704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.595773 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.596037 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.596038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.596038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.596197 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.596277 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.596454 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.596539 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:46:15.597311 25921 tablet_copy_service.cc:140] P 1d10f2df2e8e4844bf70b1aca1360a61: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 191bdb4fb70340f08dd2629ee4138671 ({username='oryx', principal='oryx/127.14.237.60@KRBTEST.COM'} at 127.0.0.1:33726)
I20260810 04:46:15.597383 25921 tablet_copy_service.cc:161] P 1d10f2df2e8e4844bf70b1aca1360a61: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 191bdb4fb70340f08dd2629ee4138671 at {username='oryx', principal='oryx/127.14.237.60@KRBTEST.COM'} at 127.0.0.1:33726: session id = 191bdb4fb70340f08dd2629ee4138671-00000000000000000000000000000000
I20260810 04:46:15.597867 25921 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 04:46:15.598381 27090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 04:46:15.599625 27090 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: tablet copy: Starting download of 0 data blocks...
I20260810 04:46:15.599779 27090 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: tablet copy: Starting download of 1 WAL segments...
I20260810 04:46:15.600600 27090 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 04:46:15.601635 25921 tablet_copy_service.cc:342] P 1d10f2df2e8e4844bf70b1aca1360a61: Request end of tablet copy session 191bdb4fb70340f08dd2629ee4138671-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.237.60@KRBTEST.COM'} at 127.0.0.1:33726
I20260810 04:46:15.601701 25921 tablet_copy_service.cc:433] P 1d10f2df2e8e4844bf70b1aca1360a61: ending tablet copy session 191bdb4fb70340f08dd2629ee4138671-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 191bdb4fb70340f08dd2629ee4138671
I20260810 04:46:15.604444 27090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:15.605661 27310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.610220 27311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.610307 27316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:15.613413 27090 server_base.cc:1034] running on GCE node
I20260810 04:46:15.613523 27090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:15.613677 27090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:15.614840 27090 hybrid_clock.cc:648] HybridClock initialized: now 1786337175614814 us; error 43 us; skew 500 ppm
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:15.616449 27090 init.cc:377] Logged in from keytab as oryx/127.14.237.60@KRBTEST.COM (short username oryx)
I20260810 04:46:15.616711 27090 webserver.cc:460] Webserver started at http://127.14.237.60:36201/ using document root <none> and password file <none>
I20260810 04:46:15.616833 27090 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.616869 27090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.617645 27090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:15.618240 27325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.618350 27090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:15.618387 27090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
uuid: "191bdb4fb70340f08dd2629ee4138671"
format_stamp: "Formatted at 2026-08-10 04:46:15 on dist-test-slave-8hhm"
server_key: "96f0c16b91da7bf111ea5be88cab9bb2"
server_key_iv: "55061c5d47415adf89f505b482484d56"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.618448 27090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.626608 27090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:15.628541 27090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:15.631788 27090 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.60:37137
I20260810 04:46:15.631924 27419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.60:37137 every 8 connection(s)
I20260810 04:46:15.632150 27090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
I20260810 04:46:15.634260 27422 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:46:15.634792 27422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Bootstrap starting.
I20260810 04:46:15.637180 27422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:46:15.637434 27422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: Bootstrap complete.
I20260810 04:46:15.637624 27422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:15.637719 27422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191bdb4fb70340f08dd2629ee4138671, State: Initialized, Role: FOLLOWER
I20260810 04:46:15.637802 27422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:15.638007 27422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:15.637981 27425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:15.638204 27425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:15.639449 27448 catalog_manager.cc:1259] Loaded cluster ID: 7b6c64b8ebc34fb79ce19293f2cd6cde
I20260810 04:46:15.639534 27448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: loading cluster ID for follower catalog manager: success
I20260810 04:46:15.641103 27448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: acquiring CA information for follower catalog manager: success
I20260810 04:46:15.641530 27448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 04:46:15.667609 27288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:15.667842 27288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:15.667879 27288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:15.669068 27288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:15.669131 27288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:15.669173 27288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:15.670698 27288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab
--principal=oryx/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:37009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=40015
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095,127.14.237.60:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:15.670969 27288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:15.671203 27288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:46:15.676298 27288 server_base.cc:1034] running on GCE node
W20260810 04:46:15.676234 27454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.676240 27455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.676234 27460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:15.676589 27288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:15.676867 27288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:15.678027 27288 hybrid_clock.cc:648] HybridClock initialized: now 1786337175678015 us; error 29 us; skew 500 ppm
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:15.680866 27288 init.cc:377] Logged in from keytab as oryx/127.14.237.1@KRBTEST.COM (short username oryx)
I20260810 04:46:15.681427 27288 webserver.cc:460] Webserver started at http://127.14.237.1:40015/ using document root <none> and password file <none>
I20260810 04:46:15.681684 27288 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.681766 27288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.683115 27288 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:15.683702 27475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.683877 27288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:15.683987 27288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "6e895a0fbcfc488284197d37e3b9f250"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "a78e9e69a1f871eb362e85e4c8760cdb"
server_key_iv: "8abcacb8481139dfc4ba62330a7705ab"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.684319 27288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.694023 27288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:15.696132 27288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:15.696286 27288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:15.696467 27288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:15.696729 27288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:15.696758 27288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.696779 27288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:15.696792 27288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.703011 27288 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:37009
I20260810 04:46:15.703065 27681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:37009 every 8 connection(s)
I20260810 04:46:15.703402 27288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.60@KRBTEST.COM
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.61@KRBTEST.COM
I20260810 04:46:15.712950 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 27288
I20260810 04:46:15.713115 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 26245
I20260810 04:46:15.718931 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.704137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:48823 (local address 127.14.237.60:37137)
0810 04:46:15.704424 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.704426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.704436 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:15.705000 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.705002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.705016 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.705079 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.705839 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.706819 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.707572 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.707861 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.707928 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.710427 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.710429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.710429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.710437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.712577 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.717836 (+ 5259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.717837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.717837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.717925 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.718219 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.718220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.718221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.718369 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.718447 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.718744 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.718837 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":63,"threads_started":1}
I20260810 04:46:15.719342 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.704101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:37137 (local address 127.14.237.1:48823)
0810 04:46:15.704285 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.704328 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.704877 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.705150 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.705154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.705525 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.705722 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.705726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.706904 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.706905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.707482 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.707484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.707950 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.707951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.708107 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.708712 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.708717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.710334 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.712692 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.712699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.712701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.713090 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.717998 (+ 4908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.717998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.717999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.718150 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.718438 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.718445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.718584 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.718721 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.718885 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260810 04:46:15.719933 27682 heartbeater.cc:347] Connected to a master server at 127.14.237.60:37137
I20260810 04:46:15.719950 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.704563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:40589 (local address 127.14.237.61:41095)
0810 04:46:15.704624 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.704626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.704737 (+ 111us) server_negotiation.cc:407] Connection header received
0810 04:46:15.704988 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.704989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.705000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.705032 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.705870 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.706605 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.707644 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.707920 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.707978 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.711893 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.711894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.711894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.711902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.713623 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.718880 (+ 5257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.718881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.718881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.718939 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.719349 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.719350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.719350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.719501 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.719550 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.719734 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.719820 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:46:15.720136 27682 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:15.720321 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.704572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:53039 (local address 127.14.237.62:37911)
0810 04:46:15.704626 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.704628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.704638 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:15.704987 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.704989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.705000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.705032 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.705851 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.706373 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.707678 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.707868 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.707951 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.710767 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.710768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.710769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.710776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.712316 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.719439 (+ 7123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.719440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.719441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.719511 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.719720 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.719721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.719721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.719862 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.719917 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.720118 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.720237 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:46:15.720336 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.704440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.14.237.1:40589)
0810 04:46:15.704660 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.704669 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.704894 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.705151 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.705155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.705706 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.705756 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.705758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.706697 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.706698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.707571 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.707572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.708002 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.708003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.708108 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.708658 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.708665 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.711748 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.718545 (+ 6797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.718546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.718547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.718791 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.719009 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.719009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.719010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.719271 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.719583 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.719584 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.719625 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.720068 (+ 443us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.720190 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":31,"thread_start_us":78,"threads_started":1}
I20260810 04:46:15.720546 27682 heartbeater.cc:511] Master 127.14.237.60:37137 requested a full tablet report, sending...
I20260810 04:46:15.720635 27684 heartbeater.cc:347] Connected to a master server at 127.14.237.61:41095
I20260810 04:46:15.720655 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.704080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.1:53039)
0810 04:46:15.704283 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.704328 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.704891 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.705151 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.705155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.705598 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.705723 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.705730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.706472 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.706473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.707568 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.707571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.708305 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.708306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.708345 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.708863 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.708867 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.710682 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.712702 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.712703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.712704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.719334 (+ 6630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.719586 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.719587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.719587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.719646 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.719950 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.719951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.719991 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.720446 (+ 455us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.720561 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260810 04:46:15.720698 27684 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:15.720768 27684 heartbeater.cc:511] Master 127.14.237.61:41095 requested a full tablet report, sending...
I20260810 04:46:15.721292 27343 ts_manager.cc:194] Registered new tserver with Master: 6e895a0fbcfc488284197d37e3b9f250 (127.14.237.1:37009)
I20260810 04:46:15.721346 26997 ts_manager.cc:194] Registered new tserver with Master: 6e895a0fbcfc488284197d37e3b9f250 (127.14.237.1:37009)
I20260810 04:46:15.721469 27683 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37911
I20260810 04:46:15.721508 27683 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:15.721573 27683 heartbeater.cc:511] Master 127.14.237.62:37911 requested a full tablet report, sending...
I20260810 04:46:15.721829 25881 ts_manager.cc:194] Re-registered known tserver with Master: 6e895a0fbcfc488284197d37e3b9f250 (127.14.237.1:37009)
I20260810 04:46:15.722339 25881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:53039
I20260810 04:46:15.724336 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:46599
--local_ip_for_outbound_sockets=127.14.237.2
--tserver_master_addrs=127.14.237.62:37911
--webserver_port=43187
--webserver_interface=127.14.237.2
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab
--principal=oryx/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095,127.14.237.60:37137 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:15.808708 27694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:15.808912 27694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:15.808935 27694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:15.810199 27694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:15.810258 27694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:15.810284 27694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:15.811897 27694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab
--principal=oryx/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:46599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=43187
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095,127.14.237.60:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:15.812163 27694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:15.812439 27694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:15.817453 27703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.817514 27709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.817454 27705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:15.817796 27694 server_base.cc:1034] running on GCE node
I20260810 04:46:15.817999 27694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:15.818310 27694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:15.819473 27694 hybrid_clock.cc:648] HybridClock initialized: now 1786337175819461 us; error 29 us; skew 500 ppm
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:15.822003 27694 init.cc:377] Logged in from keytab as oryx/127.14.237.2@KRBTEST.COM (short username oryx)
I20260810 04:46:15.822420 27694 webserver.cc:460] Webserver started at http://127.14.237.2:43187/ using document root <none> and password file <none>
I20260810 04:46:15.822647 27694 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.822729 27694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.824075 27694 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:15.824671 27719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.824847 27694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:15.824952 27694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "eb519794311a6da2b65916206bf8c8cd"
server_key_iv: "8abc0588258ef98b8bb2c97285058534"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.825338 27694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.858163 27694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:15.860862 27694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:15.861042 27694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:15.861373 27694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:15.861714 27694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:15.861773 27694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.861828 27694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:15.861855 27694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.867743 27694 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:46599
I20260810 04:46:15.867825 27950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:46599 every 8 connection(s)
I20260810 04:46:15.868192 27694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:46:15.869722 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 27694
I20260810 04:46:15.869915 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 26487
I20260810 04:46:15.883460 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:39199
--local_ip_for_outbound_sockets=127.14.237.3
--tserver_master_addrs=127.14.237.62:37911
--webserver_port=38169
--webserver_interface=127.14.237.3
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab
--principal=oryx/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095,127.14.237.60:37137 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.61@KRBTEST.COM
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.60@KRBTEST.COM
I20260810 04:46:15.892839 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.869286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.14.237.2:34299)
0810 04:46:15.869466 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.869475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.882124 (+ 12649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.882563 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.882568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.883183 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.883478 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.883484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.884623 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.884623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.885841 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.885844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.886335 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.886336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.886387 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.887105 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.887113 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.888962 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.891023 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.891031 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.891033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.891374 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.891598 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.891599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.891600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.891729 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.892063 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.892071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.892253 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.892360 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.892546 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":181,"thread_start_us":63,"threads_started":1}
I20260810 04:46:15.893404 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.869360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:34299 (local address 127.14.237.61:41095)
0810 04:46:15.869409 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.869411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.869539 (+ 128us) server_negotiation.cc:407] Connection header received
0810 04:46:15.882286 (+ 12747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.882287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.882304 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.882339 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.883822 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.884536 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.885928 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.886222 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.886285 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.889078 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.889080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.889080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.889102 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.890934 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.891463 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.891465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.891465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.891526 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.891827 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.891828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.891829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.891969 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.892035 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.893142 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.893284 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:46:15.894268 27958 heartbeater.cc:347] Connected to a master server at 127.14.237.61:41095
I20260810 04:46:15.894379 27958 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:15.894596 27958 heartbeater.cc:511] Master 127.14.237.61:41095 requested a full tablet report, sending...
I20260810 04:46:15.895181 26997 ts_manager.cc:194] Registered new tserver with Master: 5e2fc734fc3c4b2fb140b97a5a5ec651 (127.14.237.2:46599)
I20260810 04:46:15.896153 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.868969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:37137 (local address 127.14.237.2:40043)
0810 04:46:15.869394 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.869430 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.882124 (+ 12694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.882564 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.882568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.883171 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.883478 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.883484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.884515 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.884516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.885841 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.885844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.886552 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.886552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.887187 (+ 635us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.887639 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.887644 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.892058 (+ 4414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.894682 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.894683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.894684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.894937 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.895226 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.895227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.895227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.895464 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.895784 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.895784 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.895825 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.895909 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.896014 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":161,"thread_start_us":85,"threads_started":1}
I20260810 04:46:15.896224 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.869036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:40043 (local address 127.14.237.60:37137)
0810 04:46:15.869202 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.869204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.869484 (+ 280us) server_negotiation.cc:407] Connection header received
0810 04:46:15.882285 (+ 12801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.882287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.882307 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.882339 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.883605 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.884398 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.885952 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.886397 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.886455 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.892741 (+ 6286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.892743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.892743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.892752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.894540 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.895049 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.895050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.895050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.895135 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.895543 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.895544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.895544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.895691 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.895749 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.895991 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.896093 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260810 04:46:15.896836 27963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.868948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.2:33541)
0810 04:46:15.869132 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:15.869176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:15.882111 (+ 12935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:15.882575 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:15.882577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:15.883134 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:15.883478 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.883485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.884508 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.884509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.885692 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:15.885695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.886223 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.886225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:15.886383 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.887200 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:15.887204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:15.892175 (+ 4971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:15.895691 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.895691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.895692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.895902 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.896116 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:15.896116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:15.896117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:15.896174 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:15.896488 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:15.896488 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:15.896526 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 04:46:15.896583 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:46:15.896763 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":93,"thread_start_us":99,"threads_started":1}
I20260810 04:46:15.896912 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:15.868977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:33541 (local address 127.14.237.62:37911)
0810 04:46:15.869041 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:15.869043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:15.869290 (+ 247us) server_negotiation.cc:407] Connection header received
0810 04:46:15.882285 (+ 12995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:15.882288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:15.882304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:15.882338 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:15.883611 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.884372 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.885797 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:15.886085 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:15.886151 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:15.892268 (+ 6117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:15.892269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:15.892270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:15.892278 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:15.895582 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.895988 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.895989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.895989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.896049 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:15.896256 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:15.896257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:15.896258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:15.896409 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:15.896457 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:15.896684 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 04:46:15.896784 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:46:15.897312 27952 heartbeater.cc:347] Connected to a master server at 127.14.237.60:37137
I20260810 04:46:15.897363 27952 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:15.897432 27952 heartbeater.cc:511] Master 127.14.237.60:37137 requested a full tablet report, sending...
I20260810 04:46:15.897728 27343 ts_manager.cc:194] Registered new tserver with Master: 5e2fc734fc3c4b2fb140b97a5a5ec651 (127.14.237.2:46599)
I20260810 04:46:15.898216 27954 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37911
I20260810 04:46:15.898276 27954 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:15.898342 27954 heartbeater.cc:511] Master 127.14.237.62:37911 requested a full tablet report, sending...
I20260810 04:46:15.898561 25881 ts_manager.cc:194] Re-registered known tserver with Master: 5e2fc734fc3c4b2fb140b97a5a5ec651 (127.14.237.2:46599)
I20260810 04:46:15.898942 25881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:33541
W20260810 04:46:15.972244 27967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:15.972450 27967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:15.972471 27967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:15.973671 27967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:15.973716 27967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:15.973742 27967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:15.975211 27967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab
--principal=oryx/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:39199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=38169
--tserver_master_addrs=127.14.237.62:37911,127.14.237.61:41095,127.14.237.60:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:15.975454 27967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:15.975700 27967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:15.981076 27980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.981065 27976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:15.981174 27978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:15.981293 27967 server_base.cc:1034] running on GCE node
I20260810 04:46:15.981489 27967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:15.981750 27967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:15.982911 27967 hybrid_clock.cc:648] HybridClock initialized: now 1786337175982900 us; error 28 us; skew 500 ppm
Aug 10 04:46:15 dist-test-slave-8hhm krb5kdc[25793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:15.985600 27967 init.cc:377] Logged in from keytab as oryx/127.14.237.3@KRBTEST.COM (short username oryx)
I20260810 04:46:15.986025 27967 webserver.cc:460] Webserver started at http://127.14.237.3:38169/ using document root <none> and password file <none>
I20260810 04:46:15.986287 27967 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:15.986347 27967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:15.987710 27967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:15.988281 27998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.988466 27967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:15.988566 27967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "cc7e65bae65841319631e23b85e7fdf8"
format_stamp: "Formatted at 2026-08-10 04:46:14 on dist-test-slave-8hhm"
server_key: "f017f58c403e1776590a4d25e0d54943"
server_key_iv: "a93e3e7e73f9cf7beacf9240929e0806"
server_key_version: "encryptionkey@0"
I20260810 04:46:15.988903 27967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:15.997985 27967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:15.999778 27967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:15.999929 27967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:16.000157 27967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:16.000567 27967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:16.000604 27967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:16.000665 27967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:16.000699 27967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:16.007117 27967 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:39199
I20260810 04:46:16.007180 28206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:39199 every 8 connection(s)
I20260810 04:46:16.007512 27967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:16.010710 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 27967
Aug 10 04:46:16 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
Aug 10 04:46:16 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.61@KRBTEST.COM
Aug 10 04:46:16 dist-test-slave-8hhm krb5kdc[25793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337175, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.60@KRBTEST.COM
I20260810 04:46:16.021672 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.008262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:33935 (local address 127.14.237.62:37911)
0810 04:46:16.008344 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.008347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.008686 (+ 339us) server_negotiation.cc:407] Connection header received
0810 04:46:16.009900 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.009902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.009915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.009948 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.010719 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.011458 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013218 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.013558 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013602 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.018073 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.018075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.018075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.018082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.020282 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.020756 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.020757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.020757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.020816 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.021055 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.021056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.021056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.021191 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.021281 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.021496 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.021568 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 04:46:16.021764 28215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.008209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:37137 (local address 127.14.237.3:47219)
0810 04:46:16.008465 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.008510 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.009738 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.010030 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.010034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.010337 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:16.010604 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.010612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.011657 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.011658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.013119 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.013122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.013751 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.013842 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.014493 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:16.014504 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:16.017973 (+ 3469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:16.020239 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:16.020240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:16.020240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:16.020547 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:16.020778 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:16.020779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:16.020779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:16.020845 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:16.021162 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:16.021163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:16.021206 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.021335 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.021529 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":110,"thread_start_us":103,"threads_started":1}
I20260810 04:46:16.021996 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.008211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.3:33935)
0810 04:46:16.008618 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.008629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.009787 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.010031 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.010034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.010469 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:16.010605 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.010613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.011594 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.011595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.013113 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.013116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.013697 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.013837 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.014741 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:16.014746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:16.017973 (+ 3227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:16.020353 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:16.020353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:16.020354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:16.020660 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:16.020920 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:16.020920 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:16.020921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:16.020977 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:16.021333 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:16.021334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:16.021371 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.021793 (+ 422us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.021897 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":162,"thread_start_us":95,"threads_started":1}
I20260810 04:46:16.022078 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.008334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:47219 (local address 127.14.237.60:37137)
0810 04:46:16.008395 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.008397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.008600 (+ 203us) server_negotiation.cc:407] Connection header received
0810 04:46:16.009881 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.009883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.009897 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.009961 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.010738 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.011551 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013259 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.013609 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013689 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.018068 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.018069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.018070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.018077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.020154 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.020633 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.020634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.020635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.020700 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.020920 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.020921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.020921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.021059 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.021134 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.021888 (+ 754us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.021967 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:46:16.022703 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.008592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:56445 (local address 127.14.237.61:41095)
0810 04:46:16.008646 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.008648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.008708 (+ 60us) server_negotiation.cc:407] Connection header received
0810 04:46:16.009900 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.009902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.009915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.009948 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.010720 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.011457 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013240 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.013558 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013602 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.017555 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.017557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.017557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.017564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.019216 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.019719 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.019720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.019735 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.019796 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.020089 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.020090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.020091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.020226 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.020270 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.022510 (+ 2240us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.022602 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:46:16.022818 28209 heartbeater.cc:347] Connected to a master server at 127.14.237.62:37911
I20260810 04:46:16.022923 28209 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:16.023070 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.008486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.14.237.3:56445)
0810 04:46:16.008618 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.008629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.009787 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.010031 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.010034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.010473 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:16.010605 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.010613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.011619 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.011620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.013113 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.013116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.013695 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.013696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.013837 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.014613 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:16.014619 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:16.017450 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:16.019308 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:16.019314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:16.019316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:16.019624 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:16.019876 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:16.019877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:16.019877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:16.020005 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:16.020320 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:16.020326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:16.020484 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.022831 (+ 2347us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.022940 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":197,"thread_start_us":71,"threads_started":1}
I20260810 04:46:16.023113 28209 heartbeater.cc:511] Master 127.14.237.62:37911 requested a full tablet report, sending...
I20260810 04:46:16.023391 28207 heartbeater.cc:347] Connected to a master server at 127.14.237.60:37137
I20260810 04:46:16.023447 28207 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:16.023514 28207 heartbeater.cc:511] Master 127.14.237.60:37137 requested a full tablet report, sending...
I20260810 04:46:16.023686 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.012486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.14.237.62:37911)
0810 04:46:16.013389 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.013391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.013573 (+ 182us) server_negotiation.cc:407] Connection header received
0810 04:46:16.013597 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.013598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.013609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.013679 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.014617 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.015453 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.016478 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.016754 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.016808 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.020414 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.020415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.020416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.020421 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.022061 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.022483 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.022484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.022485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.022548 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.022836 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.022837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.022838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.022966 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.023066 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.023474 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.023561 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":114,"threads_started":1}
I20260810 04:46:16.024421 27343 ts_manager.cc:194] Registered new tserver with Master: cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199)
I20260810 04:46:16.024645 25881 ts_manager.cc:194] Re-registered known tserver with Master: cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199)
I20260810 04:46:16.024919 28212 heartbeater.cc:347] Connected to a master server at 127.14.237.61:41095
I20260810 04:46:16.024969 28212 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:16.025035 28212 heartbeater.cc:511] Master 127.14.237.61:41095 requested a full tablet report, sending...
I20260810 04:46:16.025162 25881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:33935
I20260810 04:46:16.026830 26997 ts_manager.cc:194] Registered new tserver with Master: cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199)
I20260810 04:46:16.025604 25880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:46:16.028126 25880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:16.028862 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.012230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.14.237.61:41095)
0810 04:46:16.015590 (+ 3360us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.015592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.015753 (+ 161us) server_negotiation.cc:407] Connection header received
0810 04:46:16.015796 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.015797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.015809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.015871 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.016671 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.020708 (+ 4037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.022616 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.023047 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.023215 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.025144 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.025146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.025146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.025152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.027276 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.027700 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.027701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.027701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.027772 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.028141 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.028142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.028142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.028326 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.028450 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.028602 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.028689 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3336,"thread_start_us":50,"threads_started":1}
I20260810 04:46:16.029552 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.013033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.14.237.60:37137)
0810 04:46:16.013328 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.013330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.015932 (+ 2602us) server_negotiation.cc:407] Connection header received
0810 04:46:16.015959 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.015960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.015973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.016053 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.016963 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.018508 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.023216 (+ 4708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.023676 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.023845 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.026069 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.026071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.026071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.026079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.027893 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.028500 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.028502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.028502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.028568 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.028823 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.028824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.028825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.028949 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.029383 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.029422 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.029496 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":58,"threads_started":1}
I20260810 04:46:16.037055 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.032687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:39199 (local address 127.0.0.1:35600)
0810 04:46:16.032790 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.032803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.032861 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.033131 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.033132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.033134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:16.033180 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.033190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.034491 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.034493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.035730 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.035733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.036777 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.036778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.036854 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.036858 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.036905 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.036961 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
I20260810 04:46:16.037388 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.032786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.14.237.3:39199)
0810 04:46:16.032970 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.032976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.032987 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:46:16.033029 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.033031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.033048 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.033131 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:16.033334 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.034369 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.035821 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.036600 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.036669 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.036740 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.037220 (+ 480us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.037309 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260810 04:46:16.037547 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.033011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:46599 (local address 127.0.0.1:44012)
0810 04:46:16.033126 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.033135 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.033174 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.033636 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.033636 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.033638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:16.033679 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.033680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.034833 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.034833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.035929 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.035931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.037312 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.037313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.037383 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.037389 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.037421 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.037461 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260810 04:46:16.037926 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.033302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.14.237.2:46599)
0810 04:46:16.033482 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.033487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.033496 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:16.033534 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.033536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.033554 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.033637 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:16.033729 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.034738 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.036009 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.037199 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.037691 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.037785 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.037811 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.037847 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260810 04:46:16.038067 28088 tablet_service.cc:1467] Processing CreateTablet for tablet 22d999f5e5494f8e8f6bfe67a783b64a (DEFAULT_TABLE table=test-table [id=ac85c90fba2e42d2a15657b779115d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:16.038323 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.032839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:37009 (local address 127.0.0.1:57596)
0810 04:46:16.032967 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.032977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.033024 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.034672 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.034673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.034675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:16.034723 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.034724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.035908 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.035909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.036969 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.036971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.037962 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.037963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.038038 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.038141 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.038190 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.038223 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":94,"thread_start_us":80,"threads_started":1}
I20260810 04:46:16.038509 28088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d999f5e5494f8e8f6bfe67a783b64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:16.038765 27819 tablet_service.cc:1467] Processing CreateTablet for tablet 22d999f5e5494f8e8f6bfe67a783b64a (DEFAULT_TABLE table=test-table [id=ac85c90fba2e42d2a15657b779115d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:16.038944 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.032941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.14.237.1:37009)
0810 04:46:16.034533 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.034538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.034548 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:16.034585 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.034587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.034601 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.034673 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:16.034798 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.035755 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.037073 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.037876 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.038057 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.038784 (+ 727us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.038818 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.038861 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":77,"threads_started":1}
I20260810 04:46:16.039108 27819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d999f5e5494f8e8f6bfe67a783b64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:16.040313 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.036196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:37137 (local address 127.0.0.1:40960)
0810 04:46:16.036347 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.036358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.036407 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.036608 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.036609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.036610 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:16.036644 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.036645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.037872 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.037873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.039518 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.039521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.040101 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.040102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.040156 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.040160 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.040201 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.040237 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260810 04:46:16.040310 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.036358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.14.237.60:37137)
0810 04:46:16.036393 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.036395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.036403 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:46:16.036490 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.036492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.036513 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.036547 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:16.036712 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.037431 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.039620 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.040014 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.040106 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.040173 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.040203 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.040240 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:46:16.041584 27363 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.10] Dedup: 1.8->[1.9-1.10]
I20260810 04:46:16.041930 27562 tablet_service.cc:1467] Processing CreateTablet for tablet 22d999f5e5494f8e8f6bfe67a783b64a (DEFAULT_TABLE table=test-table [id=ac85c90fba2e42d2a15657b779115d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:16.042095 27425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d10f2df2e8e4844bf70b1aca1360a61. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:16.042202 27425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:16.042282 27562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d999f5e5494f8e8f6bfe67a783b64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:16.043309 28280 tablet_bootstrap.cc:492] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Bootstrap starting.
I20260810 04:46:16.043955 28280 tablet_bootstrap.cc:654] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:16.043964 28274 tablet_bootstrap.cc:492] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: Bootstrap starting.
I20260810 04:46:16.044322 28280 log.cc:824] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:16.044605 28274 tablet_bootstrap.cc:654] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:16.044989 28274 log.cc:824] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:16.045245 28280 tablet_bootstrap.cc:492] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: No bootstrap required, opened a new log
I20260810 04:46:16.045353 28280 ts_tablet_manager.cc:1397] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:46:16.045380 28297 tablet_bootstrap.cc:492] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: Bootstrap starting.
I20260810 04:46:16.045893 28274 tablet_bootstrap.cc:492] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: No bootstrap required, opened a new log
I20260810 04:46:16.045969 28274 ts_tablet_manager.cc:1397] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:16.047492 28280 raft_consensus.cc:348] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.047693 28280 raft_consensus.cc:374] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:16.047747 28280 raft_consensus.cc:729] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2fc734fc3c4b2fb140b97a5a5ec651, State: Initialized, Role: FOLLOWER
I20260810 04:46:16.047885 28280 consensus_queue.cc:260] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.048147 28280 ts_tablet_manager.cc:1428] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:46:16.048297 27954 heartbeater.cc:503] Master 127.14.237.62:37911 was elected leader, sending a full tablet report...
I20260810 04:46:16.049187 28274 raft_consensus.cc:348] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.049600 28274 raft_consensus.cc:374] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:16.049677 28274 raft_consensus.cc:729] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e65bae65841319631e23b85e7fdf8, State: Initialized, Role: FOLLOWER
I20260810 04:46:16.049753 28297 tablet_bootstrap.cc:654] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:16.049810 28274 consensus_queue.cc:260] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.050094 28274 ts_tablet_manager.cc:1428] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260810 04:46:16.050081 28297 log.cc:824] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:16.050125 28209 heartbeater.cc:503] Master 127.14.237.62:37911 was elected leader, sending a full tablet report...
I20260810 04:46:16.051363 28297 tablet_bootstrap.cc:492] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: No bootstrap required, opened a new log
I20260810 04:46:16.051416 28297 ts_tablet_manager.cc:1397] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 04:46:16.052850 28297 raft_consensus.cc:348] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.052966 28297 raft_consensus.cc:374] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:16.052987 28297 raft_consensus.cc:729] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e895a0fbcfc488284197d37e3b9f250, State: Initialized, Role: FOLLOWER
I20260810 04:46:16.053056 28297 consensus_queue.cc:260] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.053318 28297 ts_tablet_manager.cc:1428] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:46:16.053331 27683 heartbeater.cc:503] Master 127.14.237.62:37911 was elected leader, sending a full tablet report...
W20260810 04:46:16.119005 27960 tablet.cc:2369] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:46:16.131917 28303 raft_consensus.cc:482] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:16.132092 28303 raft_consensus.cc:504] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.132457 28303 leader_election.cc:288] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199), 5e2fc734fc3c4b2fb140b97a5a5ec651 (127.14.237.2:46599)
I20260810 04:46:16.136579 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.132821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:46599 (local address 127.14.237.1:43475)
0810 04:46:16.132887 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.132897 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.132937 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.133140 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.133140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.133142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:16.133198 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.133200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.134164 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.134165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.135484 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.135487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.136323 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.136324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.136404 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.136407 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.136455 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.136485 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":109}
I20260810 04:46:16.136617 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.132900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:43475 (local address 127.14.237.2:46599)
0810 04:46:16.132975 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.132977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.132986 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:16.133038 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.133039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.133049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.133097 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:16.133320 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.134037 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.135581 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.136195 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.136313 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.136323 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.136482 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.136530 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:46:16.136667 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.132748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:43251 (local address 127.14.237.3:39199)
0810 04:46:16.132819 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.132821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.132832 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:46:16.132917 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.132918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.132929 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.133002 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:16.133266 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.134039 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.135587 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.136246 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.136314 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.136323 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.136547 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.136592 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:46:16.136744 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.132662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:39199 (local address 127.14.237.1:43251)
0810 04:46:16.132749 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:16.132760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:16.132851 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:16.133026 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:16.133028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:16.133029 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:16.133152 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.133154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.134164 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.134165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.135484 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:16.135487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.136365 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.136366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:16.136461 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.136465 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:16.136529 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:46:16.136572 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":59}
I20260810 04:46:16.137032 28120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d999f5e5494f8e8f6bfe67a783b64a" candidate_uuid: "6e895a0fbcfc488284197d37e3b9f250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e65bae65841319631e23b85e7fdf8" is_pre_election: true
I20260810 04:46:16.137024 27874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d999f5e5494f8e8f6bfe67a783b64a" candidate_uuid: "6e895a0fbcfc488284197d37e3b9f250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" is_pre_election: true
I20260810 04:46:16.137181 27874 raft_consensus.cc:2454] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e895a0fbcfc488284197d37e3b9f250 in term 0.
I20260810 04:46:16.137207 28120 raft_consensus.cc:2454] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e895a0fbcfc488284197d37e3b9f250 in term 0.
I20260810 04:46:16.137450 27485 leader_election.cc:302] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2fc734fc3c4b2fb140b97a5a5ec651, 6e895a0fbcfc488284197d37e3b9f250; no voters:
I20260810 04:46:16.137667 28303 raft_consensus.cc:2781] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:16.137722 28303 raft_consensus.cc:482] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:16.137746 28303 raft_consensus.cc:3037] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:16.138613 28303 raft_consensus.cc:504] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.138792 28303 leader_election.cc:288] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [CANDIDATE]: Term 1 election: Requested vote from peers cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199), 5e2fc734fc3c4b2fb140b97a5a5ec651 (127.14.237.2:46599)
I20260810 04:46:16.138947 28120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d999f5e5494f8e8f6bfe67a783b64a" candidate_uuid: "6e895a0fbcfc488284197d37e3b9f250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e65bae65841319631e23b85e7fdf8"
I20260810 04:46:16.139024 28120 raft_consensus.cc:3037] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:16.138993 27874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d999f5e5494f8e8f6bfe67a783b64a" candidate_uuid: "6e895a0fbcfc488284197d37e3b9f250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651"
I20260810 04:46:16.139254 27874 raft_consensus.cc:3037] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:16.140120 28120 raft_consensus.cc:2454] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e895a0fbcfc488284197d37e3b9f250 in term 1.
I20260810 04:46:16.140321 27485 leader_election.cc:302] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e895a0fbcfc488284197d37e3b9f250, cc7e65bae65841319631e23b85e7fdf8; no voters:
I20260810 04:46:16.140436 28303 raft_consensus.cc:2781] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:16.140589 27874 raft_consensus.cc:2454] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e895a0fbcfc488284197d37e3b9f250 in term 1.
I20260810 04:46:16.140635 28303 raft_consensus.cc:686] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 1 LEADER]: Becoming Leader. State: Replica: 6e895a0fbcfc488284197d37e3b9f250, State: Running, Role: LEADER
I20260810 04:46:16.140765 28303 consensus_queue.cc:237] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:16.141665 25880 catalog_manager.cc:5495] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e895a0fbcfc488284197d37e3b9f250 (127.14.237.1). New cstate: current_term: 1 leader_uuid: "6e895a0fbcfc488284197d37e3b9f250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } health_report { overall_health: UNKNOWN } } }
I20260810 04:46:16.142336 27278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:16.142438 27278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:16.160126 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.157586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.14.237.1:37009)
0810 04:46:16.157645 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.157647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.157657 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:16.157711 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.157712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.157730 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.157816 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:16.157950 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.158448 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.159260 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.159473 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.159553 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.159727 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:16.159841 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:16.159962 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.160006 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.160033 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:46:16.162213 27874 raft_consensus.cc:1261] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 FOLLOWER]: Refusing update from remote peer 6e895a0fbcfc488284197d37e3b9f250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:16.162248 28120 raft_consensus.cc:1261] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 1 FOLLOWER]: Refusing update from remote peer 6e895a0fbcfc488284197d37e3b9f250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:16.162633 28304 consensus_queue.cc:1035] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:16.162794 28303 consensus_queue.cc:1035] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:16.166203 28312 mvcc.cc:205] Tried to move back new op lower bound from 7316837073557381120 to 7316837073473368064. Current Snapshot: MvccSnapshot[applied={T|T < 7316837073557381120}]
I20260810 04:46:16.166532 28311 mvcc.cc:205] Tried to move back new op lower bound from 7316837073557381120 to 7316837073473368064. Current Snapshot: MvccSnapshot[applied={T|T < 7316837073557381120 or (T in {7316837073557381120})}]
I20260810 04:46:16.166935 28313 mvcc.cc:205] Tried to move back new op lower bound from 7316837073557381120 to 7316837073473368064. Current Snapshot: MvccSnapshot[applied={T|T < 7316837073557381120}]
I20260810 04:46:16.168535 25881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:16.168650 25881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:16.169920 25881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d10f2df2e8e4844bf70b1aca1360a61: Sending DeleteTablet for 3 replicas of tablet 22d999f5e5494f8e8f6bfe67a783b64a
I20260810 04:46:16.170287 28088 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d999f5e5494f8e8f6bfe67a783b64a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:16 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:35600
I20260810 04:46:16.170329 27819 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d999f5e5494f8e8f6bfe67a783b64a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:16 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:44012
I20260810 04:46:16.170598 27562 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d999f5e5494f8e8f6bfe67a783b64a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:16 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:57596
I20260810 04:46:16.170826 28332 tablet_replica.cc:330] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: stopping tablet replica
I20260810 04:46:16.170986 28332 raft_consensus.cc:2229] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:16.171073 28326 tablet_replica.cc:330] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: stopping tablet replica
I20260810 04:46:16.171175 28326 raft_consensus.cc:2229] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:16.171223 28332 raft_consensus.cc:2258] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:16.171311 28326 raft_consensus.cc:2258] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:16.171483 28331 tablet_replica.cc:330] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: stopping tablet replica
I20260810 04:46:16.171597 28331 raft_consensus.cc:2229] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:16.171703 28331 raft_consensus.cc:2258] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:16.172168 28331 ts_tablet_manager.cc:1905] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:16.172168 28332 ts_tablet_manager.cc:1905] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:16.172262 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 25818
I20260810 04:46:16.172539 28326 ts_tablet_manager.cc:1905] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:16.174085 28331 ts_tablet_manager.cc:1918] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:16.174189 28331 log.cc:1196] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/22d999f5e5494f8e8f6bfe67a783b64a
I20260810 04:46:16.174539 28331 ts_tablet_manager.cc:1939] T 22d999f5e5494f8e8f6bfe67a783b64a P 5e2fc734fc3c4b2fb140b97a5a5ec651: Deleting consensus metadata
I20260810 04:46:16.176156 28326 ts_tablet_manager.cc:1918] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:16.176690 28332 ts_tablet_manager.cc:1918] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:16.176834 28326 log.cc:1196] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/22d999f5e5494f8e8f6bfe67a783b64a
I20260810 04:46:16.176936 28332 log.cc:1196] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/22d999f5e5494f8e8f6bfe67a783b64a
I20260810 04:46:16.177615 28332 ts_tablet_manager.cc:1939] T 22d999f5e5494f8e8f6bfe67a783b64a P 6e895a0fbcfc488284197d37e3b9f250: Deleting consensus metadata
I20260810 04:46:16.177835 28326 ts_tablet_manager.cc:1939] T 22d999f5e5494f8e8f6bfe67a783b64a P cc7e65bae65841319631e23b85e7fdf8: Deleting consensus metadata
I20260810 04:46:16.183190 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.171460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.14.237.60:37137)
0810 04:46:16.171500 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.171503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.175100 (+ 3597us) server_negotiation.cc:407] Connection header received
0810 04:46:16.175217 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.175218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.175231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.175268 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.176245 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.176953 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.178271 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.178552 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.178602 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.180287 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.180288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.180288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.180296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.181880 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.182303 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.182304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.182304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.182374 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.182606 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.182607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.182608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.182734 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.182866 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.182987 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.183077 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 04:46:16.188315 27724 connection.cc:535] server connection from 127.0.0.1:44012 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 04:46:16.188416 27485 connection.cc:535] server connection from 127.0.0.1:57596 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 04:46:16.188629 28003 connection.cc:535] server connection from 127.0.0.1:35600 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:46:16.198508 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.190465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.14.237.60:37137)
0810 04:46:16.190503 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.190505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.190644 (+ 139us) server_negotiation.cc:407] Connection header received
0810 04:46:16.190736 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.190737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.190747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.190777 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.191525 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.192158 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.193275 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.193581 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.193650 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.195136 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.195137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.195137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.195145 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.196914 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.197424 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.197426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.197426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.197493 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.197872 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.197873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.197874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.198009 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.198087 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.198312 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.198394 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:46:16.198510 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:16.190614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.14.237.61:41095)
0810 04:46:16.190662 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:16.190664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:16.190683 (+ 19us) server_negotiation.cc:407] Connection header received
0810 04:46:16.190713 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:16.190714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:16.190727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:16.190789 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:16.191692 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.192501 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.193481 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:16.193782 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:16.193865 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:16.195343 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:16.195344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:16.195345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:16.195353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:16.197078 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.197574 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.197576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.197576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.197643 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:16.197846 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:16.197847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:16.197847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:16.197987 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:16.198102 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:16.198232 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 04:46:16.198322 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:46:17.167167 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.166797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.3:37729)
0810 04:46:17.167019 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.167120 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
W20260810 04:46:17.167419 28209 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:37911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
I20260810 04:46:17.168360 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.167926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.2:56569)
0810 04:46:17.168092 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.168284 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
W20260810 04:46:17.168708 27954 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:37911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
I20260810 04:46:17.168787 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.168468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.1:50521)
0810 04:46:17.168660 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.168743 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":49,"threads_started":1}
W20260810 04:46:17.168959 27683 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:37911 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
I20260810 04:46:17.670334 28349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1d10f2df2e8e4844bf70b1aca1360a61)
I20260810 04:46:17.670480 28349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:17.670719 28349 leader_election.cc:288] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d10f2df2e8e4844bf70b1aca1360a61 (127.14.237.62:37911), 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60:37137)
I20260810 04:46:17.671159 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.670889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:34202)
0810 04:46:17.671001 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.671070 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":46,"threads_started":1}
I20260810 04:46:17.671257 28358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1d10f2df2e8e4844bf70b1aca1360a61)
W20260810 04:46:17.671306 26972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
I20260810 04:46:17.671378 28358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:17.671676 28358 leader_election.cc:288] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d10f2df2e8e4844bf70b1aca1360a61 (127.14.237.62:37911), 02f9304665a3462c8bdb206d08d2a0b1 (127.14.237.61:41095)
I20260810 04:46:17.671871 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.671740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:34206)
0810 04:46:17.671782 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.671821 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 04:46:17.671957 26972 leader_election.cc:334] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d10f2df2e8e4844bf70b1aca1360a61 (127.14.237.62:37911): Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
I20260810 04:46:17.672142 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.671756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:34220)
0810 04:46:17.672058 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.672104 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":91,"threads_started":1}
W20260810 04:46:17.672252 27331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
I20260810 04:46:17.672901 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.672774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:34236)
0810 04:46:17.672839 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.672874 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 04:46:17.673025 27331 leader_election.cc:334] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d10f2df2e8e4844bf70b1aca1360a61 (127.14.237.62:37911): Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
I20260810 04:46:17.674813 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.671062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.14.237.60:37137)
0810 04:46:17.671185 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:17.671187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:17.671196 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:17.671226 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:17.671227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:17.671239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:17.671507 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:17.671545 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.672343 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.673954 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.674477 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.674551 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.674564 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:17.674721 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 04:46:17.674754 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260810 04:46:17.674898 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.670898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:37137 (local address 127.0.0.1:40988)
0810 04:46:17.671003 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.671013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:17.671072 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:17.671350 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:17.671352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:17.671355 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:17.671426 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.671428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.672435 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.672436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.673848 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.673851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.674560 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.674561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.674641 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.674645 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:17.674707 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 04:46:17.674807 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":42,"thread_start_us":44,"threads_started":1}
I20260810 04:46:17.675140 27363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02f9304665a3462c8bdb206d08d2a0b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "191bdb4fb70340f08dd2629ee4138671" is_pre_election: true
I20260810 04:46:17.675239 27363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f9304665a3462c8bdb206d08d2a0b1 in term 1.
I20260810 04:46:17.675390 26970 leader_election.cc:302] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02f9304665a3462c8bdb206d08d2a0b1, 191bdb4fb70340f08dd2629ee4138671; no voters: 1d10f2df2e8e4844bf70b1aca1360a61
I20260810 04:46:17.675500 28349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 04:46:17.675550 28349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1d10f2df2e8e4844bf70b1aca1360a61)
I20260810 04:46:17.675570 28349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:46:17.675575 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.671900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:41095 (local address 127.0.0.1:44462)
0810 04:46:17.672037 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.672048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:17.672101 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:17.672332 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:17.672333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:17.672335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:17.672407 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.672409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.673359 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.673360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.674551 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.674553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.675333 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.675334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.675400 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.675403 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:17.675457 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:46:17.675500 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260810 04:46:17.675833 28363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.672033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.14.237.61:41095)
0810 04:46:17.672200 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:17.672202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:17.672209 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:46:17.672232 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:17.672233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:17.672243 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:17.672473 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:17.672502 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.673287 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.674623 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.675207 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.675675 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.675689 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:17.675713 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 04:46:17.675773 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260810 04:46:17.676002 27010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "191bdb4fb70340f08dd2629ee4138671" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "02f9304665a3462c8bdb206d08d2a0b1" is_pre_election: true
I20260810 04:46:17.676360 28349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:17.676489 27010 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 191bdb4fb70340f08dd2629ee4138671 in current term 2: Already voted for candidate 02f9304665a3462c8bdb206d08d2a0b1 in this term.
I20260810 04:46:17.676672 27363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02f9304665a3462c8bdb206d08d2a0b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "191bdb4fb70340f08dd2629ee4138671"
I20260810 04:46:17.676687 27329 leader_election.cc:302] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191bdb4fb70340f08dd2629ee4138671; no voters: 02f9304665a3462c8bdb206d08d2a0b1, 1d10f2df2e8e4844bf70b1aca1360a61
I20260810 04:46:17.676790 27363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:46:17.676493 28349 leader_election.cc:288] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [CANDIDATE]: Term 2 election: Requested vote from peers 1d10f2df2e8e4844bf70b1aca1360a61 (127.14.237.62:37911), 191bdb4fb70340f08dd2629ee4138671 (127.14.237.60:37137)
I20260810 04:46:17.677040 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.676561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:34260)
0810 04:46:17.677001 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.677026 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430}
I20260810 04:46:17.677323 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.677257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:34266)
0810 04:46:17.677282 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.677302 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260810 04:46:17.677388 26972 leader_election.cc:334] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1d10f2df2e8e4844bf70b1aca1360a61 (127.14.237.62:37911): Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
I20260810 04:46:17.677691 27363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f9304665a3462c8bdb206d08d2a0b1 in term 2.
I20260810 04:46:17.677801 28358 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:46:17.677865 26970 leader_election.cc:302] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02f9304665a3462c8bdb206d08d2a0b1, 191bdb4fb70340f08dd2629ee4138671; no voters: 1d10f2df2e8e4844bf70b1aca1360a61
I20260810 04:46:17.677965 28349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:46:17.678082 28349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [term 2 LEADER]: Becoming Leader. State: Replica: 02f9304665a3462c8bdb206d08d2a0b1, State: Running, Role: LEADER
I20260810 04:46:17.678167 28349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } }
I20260810 04:46:17.678520 28368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02f9304665a3462c8bdb206d08d2a0b1. Latest consensus state: current_term: 2 leader_uuid: "02f9304665a3462c8bdb206d08d2a0b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:17.678604 28368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:17.678778 28371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:17.679159 28371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:17.679270 28371 catalog_manager.cc:1259] Loaded cluster ID: 7b6c64b8ebc34fb79ce19293f2cd6cde
I20260810 04:46:17.679302 28371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:17.679474 28371 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:17.679567 28371 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Loaded TSK: 0
I20260810 04:46:17.679746 28371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:46:17.732839 26993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:46:17.734845 26993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:17.736017 27363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [term 2 FOLLOWER]: Refusing update from remote peer 02f9304665a3462c8bdb206d08d2a0b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 04:46:17.736050 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.735938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:34278)
0810 04:46:17.735994 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.736025 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:46:17.736387 28349 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 04:46:17.736430 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.736313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.0.0.1:34288)
0810 04:46:17.736372 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.736403 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 04:46:17.736555 26972 consensus_peers.cc:487] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 -> Peer 1d10f2df2e8e4844bf70b1aca1360a61 (127.14.237.62:37911): Couldn't send request to peer 1d10f2df2e8e4844bf70b1aca1360a61. Status: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 04:46:17.737033 28358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02f9304665a3462c8bdb206d08d2a0b1. Latest consensus state: current_term: 2 leader_uuid: "02f9304665a3462c8bdb206d08d2a0b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:17.737131 28358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:17.737847 28358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "02f9304665a3462c8bdb206d08d2a0b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:17.737967 28358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191bdb4fb70340f08dd2629ee4138671 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:46:17.737684 28368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "02f9304665a3462c8bdb206d08d2a0b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1d10f2df2e8e4844bf70b1aca1360a61" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37911 } } peers { permanent_uuid: "02f9304665a3462c8bdb206d08d2a0b1" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 41095 } attrs { promote: false } } peers { permanent_uuid: "191bdb4fb70340f08dd2629ee4138671" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 37137 } attrs { promote: false } } }
I20260810 04:46:17.738101 28368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:17.743088 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.739711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:37009 (local address 127.0.0.1:57610)
0810 04:46:17.739759 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.739770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:17.739809 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:17.740098 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:17.740099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:17.740101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:17.740153 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.740154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.741083 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.741084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.742201 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.742203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.742849 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.742850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.742907 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.742911 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:17.742965 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:46:17.743005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:46:17.743009 28381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.739895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:46599 (local address 127.0.0.1:44014)
0810 04:46:17.740217 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.740229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:17.740273 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:17.740482 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:17.740482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:17.740484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:17.740528 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.740529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.741197 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.741198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.742077 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.742080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.742784 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.742785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.742857 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.742862 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:46:17.742890 (+ 28us) client_negotiation.cc:240] Negotiation successful
0810 04:46:17.742930 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":20,"thread_start_us":78,"threads_started":1}
I20260810 04:46:17.743214 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.739966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.14.237.2:46599)
0810 04:46:17.740345 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:17.740349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:17.740363 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:46:17.740384 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:17.740384 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:17.740394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:17.740465 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:17.740616 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.741114 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.742174 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.742686 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.743054 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.743069 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:17.743090 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 04:46:17.743129 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":55,"threads_started":1}
I20260810 04:46:17.743394 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.739819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:39199 (local address 127.0.0.1:35612)
0810 04:46:17.740060 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.740069 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:17.740149 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:17.740526 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:17.740526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:17.740528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:17.740570 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.740571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.741411 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.741412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.742448 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:17.742449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.743203 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.743204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:17.743266 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.743269 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:17.743301 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 04:46:17.743339 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260810 04:46:17.743577 27819 tablet_service.cc:1467] Processing CreateTablet for tablet 6907e60b239f48dab1f8c77b5b7aa865 (DEFAULT_TABLE table=test-table [id=7fbb8c9e72d24225b930d18af9d6dbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:17.743669 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.740224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.14.237.3:39199)
0810 04:46:17.740391 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:17.740394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:17.740401 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:46:17.740424 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:17.740425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:17.740435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:17.740520 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:17.740626 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.741332 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.742531 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.743112 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.743526 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.743541 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:17.743561 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 04:46:17.743603 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
I20260810 04:46:17.743727 27819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6907e60b239f48dab1f8c77b5b7aa865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:17.743108 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.739789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.14.237.1:37009)
0810 04:46:17.739919 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:17.739921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:17.739930 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:17.739963 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:17.739964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:17.739991 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:17.740220 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:17.740267 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.741000 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.742294 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:17.742769 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:17.742852 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:17.742866 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:17.742979 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:46:17.743019 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260810 04:46:17.743932 28088 tablet_service.cc:1467] Processing CreateTablet for tablet 6907e60b239f48dab1f8c77b5b7aa865 (DEFAULT_TABLE table=test-table [id=7fbb8c9e72d24225b930d18af9d6dbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:17.744059 28088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6907e60b239f48dab1f8c77b5b7aa865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:17.744040 27562 tablet_service.cc:1467] Processing CreateTablet for tablet 6907e60b239f48dab1f8c77b5b7aa865 (DEFAULT_TABLE table=test-table [id=7fbb8c9e72d24225b930d18af9d6dbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:17.744163 27562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6907e60b239f48dab1f8c77b5b7aa865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:17.745118 28390 tablet_bootstrap.cc:492] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: Bootstrap starting.
I20260810 04:46:17.745550 28391 tablet_bootstrap.cc:492] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: Bootstrap starting.
I20260810 04:46:17.745599 28390 tablet_bootstrap.cc:654] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:17.745962 28391 tablet_bootstrap.cc:654] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:17.746263 28394 tablet_bootstrap.cc:492] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250: Bootstrap starting.
I20260810 04:46:17.746364 28390 tablet_bootstrap.cc:492] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: No bootstrap required, opened a new log
I20260810 04:46:17.746410 28390 ts_tablet_manager.cc:1397] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:17.746500 28390 raft_consensus.cc:348] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:17.746549 28390 raft_consensus.cc:374] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:17.746562 28390 raft_consensus.cc:729] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2fc734fc3c4b2fb140b97a5a5ec651, State: Initialized, Role: FOLLOWER
I20260810 04:46:17.746618 28390 consensus_queue.cc:260] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:17.746660 28394 tablet_bootstrap.cc:654] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:17.746793 28390 ts_tablet_manager.cc:1428] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:17.747030 28391 tablet_bootstrap.cc:492] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: No bootstrap required, opened a new log
I20260810 04:46:17.747084 28391 ts_tablet_manager.cc:1397] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:17.747217 28391 raft_consensus.cc:348] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:17.747293 28391 raft_consensus.cc:374] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:17.747316 28391 raft_consensus.cc:729] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7e65bae65841319631e23b85e7fdf8, State: Initialized, Role: FOLLOWER
I20260810 04:46:17.747385 28391 consensus_queue.cc:260] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:17.747527 28394 tablet_bootstrap.cc:492] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250: No bootstrap required, opened a new log
I20260810 04:46:17.747573 28394 ts_tablet_manager.cc:1397] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:17.747575 28391 ts_tablet_manager.cc:1428] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:17.747699 28394 raft_consensus.cc:348] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:17.747747 27958 heartbeater.cc:503] Master 127.14.237.61:41095 was elected leader, sending a full tablet report...
I20260810 04:46:17.747788 28394 raft_consensus.cc:374] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:17.747808 28394 raft_consensus.cc:729] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e895a0fbcfc488284197d37e3b9f250, State: Initialized, Role: FOLLOWER
I20260810 04:46:17.747848 28394 consensus_queue.cc:260] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:17.747977 28394 ts_tablet_manager.cc:1428] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:17.748718 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.747038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.2:59771)
0810 04:46:17.748618 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.748674 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1454,"thread_start_us":89,"threads_started":1}
I20260810 04:46:17.748899 27684 heartbeater.cc:503] Master 127.14.237.61:41095 was elected leader, sending a full tablet report...
I20260810 04:46:17.749038 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.748309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.3:50069)
0810 04:46:17.748954 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.748990 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":82,"threads_started":1}
I20260810 04:46:17.749158 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.748987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.2:52453)
0810 04:46:17.749111 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.749136 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260810 04:46:17.749408 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.748183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.1:58379)
0810 04:46:17.749335 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.749368 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1128,"thread_start_us":80,"threads_started":1}
I20260810 04:46:17.749733 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:17.749642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.1:35839)
0810 04:46:17.749692 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:17.749716 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 04:46:17.760041 28214 tablet.cc:2369] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:46:17.955561 27687 tablet.cc:2369] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:46:18.001184 28398 raft_consensus.cc:482] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:18.001320 28398 raft_consensus.cc:504] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:18.001643 28398 leader_election.cc:288] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e895a0fbcfc488284197d37e3b9f250 (127.14.237.1:37009), cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199)
I20260810 04:46:18.004665 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.002061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:43581 (local address 127.14.237.3:39199)
0810 04:46:18.002113 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:18.002114 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:18.002140 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:46:18.002217 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:18.002218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:18.002229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:18.002288 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:18.002454 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.002930 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.003925 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.004295 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.004352 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:18.004360 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:18.004546 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 04:46:18.004592 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:46:18.004668 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.001984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:39199 (local address 127.14.237.2:43581)
0810 04:46:18.002067 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.002077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:18.002128 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:18.002310 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:18.002311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:18.002313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:18.002380 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:18.002382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.003016 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.003017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:18.003826 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:18.003828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.004384 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.004385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:18.004459 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:18.004462 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:18.004516 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:46:18.004547 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":2}
I20260810 04:46:18.004735 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.002001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:38739 (local address 127.14.237.1:37009)
0810 04:46:18.002065 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:18.002067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:18.002081 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:46:18.002251 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:18.002252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:18.002262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:18.002337 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:18.002515 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.002951 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.003959 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.004311 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.004403 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:18.004412 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:18.004602 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 04:46:18.004646 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:46:18.004694 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.001868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:37009 (local address 127.14.237.2:38739)
0810 04:46:18.001951 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.001965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:18.002063 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:18.002361 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:18.002362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:18.002363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:18.002410 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:18.002427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.003064 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.003065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:18.003831 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:18.003833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.004427 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.004428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:18.004495 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:18.004517 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 04:46:18.004577 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:46:18.004613 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":20}
I20260810 04:46:18.004948 28120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6907e60b239f48dab1f8c77b5b7aa865" candidate_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e65bae65841319631e23b85e7fdf8" is_pre_election: true
I20260810 04:46:18.005028 28120 raft_consensus.cc:2454] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e2fc734fc3c4b2fb140b97a5a5ec651 in term 0.
I20260810 04:46:18.005059 27595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6907e60b239f48dab1f8c77b5b7aa865" candidate_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e895a0fbcfc488284197d37e3b9f250" is_pre_election: true
I20260810 04:46:18.005179 27726 leader_election.cc:302] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2fc734fc3c4b2fb140b97a5a5ec651, cc7e65bae65841319631e23b85e7fdf8; no voters:
I20260810 04:46:18.005231 27595 raft_consensus.cc:2454] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e2fc734fc3c4b2fb140b97a5a5ec651 in term 0.
I20260810 04:46:18.005322 28398 raft_consensus.cc:2781] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:18.005384 28398 raft_consensus.cc:482] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:18.005407 28398 raft_consensus.cc:3037] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:18.006067 28398 raft_consensus.cc:504] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:18.006224 28398 leader_election.cc:288] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [CANDIDATE]: Term 1 election: Requested vote from peers 6e895a0fbcfc488284197d37e3b9f250 (127.14.237.1:37009), cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199)
I20260810 04:46:18.006415 27595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6907e60b239f48dab1f8c77b5b7aa865" candidate_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e895a0fbcfc488284197d37e3b9f250"
I20260810 04:46:18.006460 28120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6907e60b239f48dab1f8c77b5b7aa865" candidate_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7e65bae65841319631e23b85e7fdf8"
I20260810 04:46:18.006521 28120 raft_consensus.cc:3037] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:18.006513 27595 raft_consensus.cc:3037] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:18.007145 28120 raft_consensus.cc:2454] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e2fc734fc3c4b2fb140b97a5a5ec651 in term 1.
I20260810 04:46:18.007210 27595 raft_consensus.cc:2454] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e2fc734fc3c4b2fb140b97a5a5ec651 in term 1.
I20260810 04:46:18.008283 27726 leader_election.cc:302] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2fc734fc3c4b2fb140b97a5a5ec651, 6e895a0fbcfc488284197d37e3b9f250; no voters:
I20260810 04:46:18.008420 28398 raft_consensus.cc:2781] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:18.008605 28398 raft_consensus.cc:686] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 LEADER]: Becoming Leader. State: Replica: 5e2fc734fc3c4b2fb140b97a5a5ec651, State: Running, Role: LEADER
I20260810 04:46:18.008697 28398 consensus_queue.cc:237] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } }
I20260810 04:46:18.008925 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.008826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.2:39211)
0810 04:46:18.008871 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.008897 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 04:46:18.009533 26993 catalog_manager.cc:5495] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e2fc734fc3c4b2fb140b97a5a5ec651 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e2fc734fc3c4b2fb140b97a5a5ec651" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46599 } health_report { overall_health: HEALTHY } } }
I20260810 04:46:18.026021 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.023569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.14.237.2:46599)
0810 04:46:18.023639 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:18.023641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:18.023650 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:18.023686 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:18.023687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:18.023708 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:18.023806 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:18.023937 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.024404 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.025208 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:18.025441 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:18.025527 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:18.025684 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:18.025793 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:18.025868 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:18.025923 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 04:46:18.025942 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:46:18.027172 28120 raft_consensus.cc:1261] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 1 FOLLOWER]: Refusing update from remote peer 5e2fc734fc3c4b2fb140b97a5a5ec651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:18.027318 27595 raft_consensus.cc:1261] T 6907e60b239f48dab1f8c77b5b7aa865 P 6e895a0fbcfc488284197d37e3b9f250 [term 1 FOLLOWER]: Refusing update from remote peer 5e2fc734fc3c4b2fb140b97a5a5ec651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:18.027464 28398 consensus_queue.cc:1035] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7e65bae65841319631e23b85e7fdf8" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:18.027594 28409 consensus_queue.cc:1035] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e895a0fbcfc488284197d37e3b9f250" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 37009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:18.028380 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.028267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.1:52825)
0810 04:46:18.028312 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.028343 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 04:46:18.028486 28212 heartbeater.cc:503] Master 127.14.237.61:41095 was elected leader, sending a full tablet report...
I20260810 04:46:18.028878 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.028741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.3:58449)
0810 04:46:18.028816 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.028849 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 04:46:18.029742 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.029602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.2:36545)
0810 04:46:18.029680 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.029709 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 04:46:18.029924 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.029812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.1:49013)
0810 04:46:18.029873 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.029899 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 04:46:18.030138 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.030052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.2:42331)
0810 04:46:18.030086 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.030115 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260810 04:46:18.030534 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:18.030436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:37911 (local address 127.14.237.3:41483)
0810 04:46:18.030482 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.030509 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:37911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:46:18.032604 26993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:18.032709 26993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:18.034039 26993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02f9304665a3462c8bdb206d08d2a0b1: Sending DeleteTablet for 3 replicas of tablet 6907e60b239f48dab1f8c77b5b7aa865
I20260810 04:46:18.034392 28088 tablet_service.cc:1514] Processing DeleteTablet for tablet 6907e60b239f48dab1f8c77b5b7aa865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:18 UTC) from {username='oryx', principal='oryx/127.14.237.61@KRBTEST.COM'} at 127.0.0.1:35612
I20260810 04:46:18.034402 27562 tablet_service.cc:1514] Processing DeleteTablet for tablet 6907e60b239f48dab1f8c77b5b7aa865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:18 UTC) from {username='oryx', principal='oryx/127.14.237.61@KRBTEST.COM'} at 127.0.0.1:57610
I20260810 04:46:18.034431 27819 tablet_service.cc:1514] Processing DeleteTablet for tablet 6907e60b239f48dab1f8c77b5b7aa865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:18 UTC) from {username='oryx', principal='oryx/127.14.237.61@KRBTEST.COM'} at 127.0.0.1:44014
I20260810 04:46:18.034708 28432 tablet_replica.cc:330] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: stopping tablet replica
I20260810 04:46:18.034706 28434 tablet_replica.cc:330] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: stopping tablet replica
I20260810 04:46:18.034772 28434 raft_consensus.cc:2229] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:18.034857 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 27288
I20260810 04:46:18.044710 28432 raft_consensus.cc:2229] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:18.044710 28434 raft_consensus.cc:2258] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:18.044906 28432 raft_consensus.cc:2258] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:18.045127 28434 ts_tablet_manager.cc:1905] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:18.045364 28432 ts_tablet_manager.cc:1905] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:18.046977 28434 ts_tablet_manager.cc:1918] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:18.046995 28432 ts_tablet_manager.cc:1918] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:18.047052 28434 log.cc:1196] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/6907e60b239f48dab1f8c77b5b7aa865
I20260810 04:46:18.047183 28432 log.cc:1196] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/6907e60b239f48dab1f8c77b5b7aa865
I20260810 04:46:18.047313 28434 ts_tablet_manager.cc:1939] T 6907e60b239f48dab1f8c77b5b7aa865 P 5e2fc734fc3c4b2fb140b97a5a5ec651: Deleting consensus metadata
I20260810 04:46:18.047534 28432 ts_tablet_manager.cc:1939] T 6907e60b239f48dab1f8c77b5b7aa865 P cc7e65bae65841319631e23b85e7fdf8: Deleting consensus metadata
I20260810 04:46:18.047715 26972 catalog_manager.cc:4841] TS 5e2fc734fc3c4b2fb140b97a5a5ec651 (127.14.237.2:46599): tablet 6907e60b239f48dab1f8c77b5b7aa865 (table test-table [id=7fbb8c9e72d24225b930d18af9d6dbf9]) successfully deleted
I20260810 04:46:18.047921 26972 catalog_manager.cc:4841] TS cc7e65bae65841319631e23b85e7fdf8 (127.14.237.3:39199): tablet 6907e60b239f48dab1f8c77b5b7aa865 (table test-table [id=7fbb8c9e72d24225b930d18af9d6dbf9]) successfully deleted
W20260810 04:46:18.055984 28351 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:18.055821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:37009 (local address 127.0.0.1:57618)
0810 04:46:18.055881 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:18.055912 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:37009: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:46:18.056125 26972 catalog_manager.cc:4584] TS 6e895a0fbcfc488284197d37e3b9f250 (127.14.237.1:37009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6907e60b239f48dab1f8c77b5b7aa865: Network error: Client connection negotiation failed: client connection to 127.14.237.1:37009: connect: Connection reset by peer (error 104)
I20260810 04:46:18.056252 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 27694
I20260810 04:46:18.067174 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 27967
I20260810 04:46:18.080210 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 26761
I20260810 04:46:18.092557 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 27090
2026-08-10T04:46:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5133 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:18 dist-test-slave-8hhm krb5kdc[28439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:18 dist-test-slave-8hhm krb5kdc[28439](info): set up 2 sockets
Aug 10 04:46:18 dist-test-slave-8hhm krb5kdc[28439](info): commencing operation
krb5kdc: starting...
W20260810 04:46:19.361330 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:19 dist-test-slave-8hhm krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.62.keytab.
Entry for principal oryx/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.62.keytab.
I20260810 04:46:19.485652 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:42627
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.62.keytab
--principal=oryx/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:42627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:19.570487 28464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:19.570747 28464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:19.570770 28464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:19.571892 28464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:19.571925 28464 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:46:19.571938 28464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:19.571951 28464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:19.571964 28464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:19.571975 28464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:19.573529 28464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:42627
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.62.keytab
--principal=oryx/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:42627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:19.573756 28464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:19.573928 28464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:46:19.578871 28464 server_base.cc:1034] running on GCE node
W20260810 04:46:19.578855 28472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:19.578830 28471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:19.578835 28476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:19.579303 28464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:19.579591 28464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:19.580735 28464 hybrid_clock.cc:648] HybridClock initialized: now 1786337179580710 us; error 39 us; skew 500 ppm
Aug 10 04:46:19 dist-test-slave-8hhm krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337179, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:19.587893 28464 init.cc:377] Logged in from keytab as oryx/127.14.237.62@KRBTEST.COM (short username oryx)
I20260810 04:46:19.588380 28464 webserver.cc:460] Webserver started at http://127.14.237.62:38325/ using document root <none> and password file <none>
I20260810 04:46:19.588604 28464 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:19.588672 28464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:19.588827 28464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:19.590229 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "c47f9724d3964122b46d3ffc2402cb65"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "3c344f7bc6fb707064c4b190fcdd333c"
server_key_iv: "07190d519ba18ef47bf2a34d27743aa4"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.590638 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "c47f9724d3964122b46d3ffc2402cb65"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "3c344f7bc6fb707064c4b190fcdd333c"
server_key_iv: "07190d519ba18ef47bf2a34d27743aa4"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.592077 28464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:19.592890 28489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:19.593081 28464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:19.593189 28464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "c47f9724d3964122b46d3ffc2402cb65"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "3c344f7bc6fb707064c4b190fcdd333c"
server_key_iv: "07190d519ba18ef47bf2a34d27743aa4"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.593334 28464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:19.605044 28464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:19.606949 28464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:19.607106 28464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:19.611975 28464 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:42627
I20260810 04:46:19.612020 28610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:42627 every 8 connection(s)
I20260810 04:46:19.612361 28464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:19.612883 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 28464
I20260810 04:46:19.613013 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:19.613174 28612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:19.613266 15284 external_mini_cluster.cc:1428] Setting key 161e6551ecd15a5a4eee9bbad6f71916
I20260810 04:46:19.616124 28612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: Bootstrap starting.
I20260810 04:46:19.616755 28612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:19.617040 28612 log.cc:824] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:19.617731 28612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: No bootstrap required, opened a new log
Aug 10 04:46:19 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:19.619817 28612 raft_consensus.cc:348] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f9724d3964122b46d3ffc2402cb65" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42627 } }
I20260810 04:46:19.619990 28612 raft_consensus.cc:374] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:19.620043 28612 raft_consensus.cc:729] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47f9724d3964122b46d3ffc2402cb65, State: Initialized, Role: FOLLOWER
I20260810 04:46:19.620199 28612 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f9724d3964122b46d3ffc2402cb65" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42627 } }
I20260810 04:46:19.620296 28612 raft_consensus.cc:388] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:19.620343 28612 raft_consensus.cc:482] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:19.620409 28612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:19.620910 28612 raft_consensus.cc:504] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f9724d3964122b46d3ffc2402cb65" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42627 } }
I20260810 04:46:19.620995 28612 leader_election.cc:302] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c47f9724d3964122b46d3ffc2402cb65; no voters:
I20260810 04:46:19.621244 28612 leader_election.cc:288] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:19.621503 28612 sys_catalog.cc:564] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:19.621562 28627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:19.621732 28627 raft_consensus.cc:686] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [term 1 LEADER]: Becoming Leader. State: Replica: c47f9724d3964122b46d3ffc2402cb65, State: Running, Role: LEADER
I20260810 04:46:19.621822 28627 consensus_queue.cc:237] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f9724d3964122b46d3ffc2402cb65" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42627 } }
I20260810 04:46:19.622161 28627 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c47f9724d3964122b46d3ffc2402cb65. Latest consensus state: current_term: 1 leader_uuid: "c47f9724d3964122b46d3ffc2402cb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f9724d3964122b46d3ffc2402cb65" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42627 } } }
I20260810 04:46:19.622239 28627 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:19.622206 28629 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c47f9724d3964122b46d3ffc2402cb65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47f9724d3964122b46d3ffc2402cb65" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 42627 } } }
I20260810 04:46:19.622308 28629 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65 [sys.catalog]: This master's current role is: LEADER
W20260810 04:46:19.623333 28650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:46:19.623430 28650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:46:19.623675 28642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:19.623857 28642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:19.624212 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.613822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.14.237.62:42627)
0810 04:46:19.614029 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:19.614039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:19.614074 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:46:19.614631 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:19.614638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:19.614693 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:19.614853 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:19.615492 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.616443 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.617326 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.617722 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.617840 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.620364 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:19.620369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:19.620369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:19.620382 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:19.622531 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.622903 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.622906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.622906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.623004 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.623256 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.623257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.623257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.623619 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:19.623800 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:19.623911 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 04:46:19.624046 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260810 04:46:19.626176 28642 catalog_manager.cc:1347] Generated new cluster ID: f71e18ba38c443079732b955871c2060
I20260810 04:46:19.626216 28642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:19.644331 28642 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:19.644861 28642 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:19.649453 28642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: Generated new TSK 0
I20260810 04:46:19.649652 28642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab.
Entry for principal oryx/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab.
I20260810 04:46:19.681977 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:42627
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab
--principal=oryx/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:19.763821 28661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:19.764020 28661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:19.764045 28661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:19.765177 28661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:19.765232 28661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:19.765254 28661 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:46:19.765308 28661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:19.766773 28661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab
--principal=oryx/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:42627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:19.767010 28661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:19.767257 28661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:46:19.772282 28661 server_base.cc:1034] running on GCE node
W20260810 04:46:19.772240 28671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:19.772257 28673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:19.772236 28676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:19.772754 28661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:19.773051 28661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:19.774192 28661 hybrid_clock.cc:648] HybridClock initialized: now 1786337179774173 us; error 32 us; skew 500 ppm
Aug 10 04:46:19 dist-test-slave-8hhm krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337179, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:19.776811 28661 init.cc:377] Logged in from keytab as oryx/127.14.237.1@KRBTEST.COM (short username oryx)
I20260810 04:46:19.777325 28661 webserver.cc:460] Webserver started at http://127.14.237.1:38717/ using document root <none> and password file <none>
I20260810 04:46:19.777534 28661 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:19.777575 28661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:19.777662 28661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:19.778630 28661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "e243ef1700264f1f986d1875c61e5fef"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "8e9e56dbdbf671bfa33ab7c055474212"
server_key_iv: "a833f36ae4128df7fef8be54965d56f7"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.778930 28661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "e243ef1700264f1f986d1875c61e5fef"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "8e9e56dbdbf671bfa33ab7c055474212"
server_key_iv: "a833f36ae4128df7fef8be54965d56f7"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.780153 28661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:19.780812 28684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:19.780979 28661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:19.781085 28661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "e243ef1700264f1f986d1875c61e5fef"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "8e9e56dbdbf671bfa33ab7c055474212"
server_key_iv: "a833f36ae4128df7fef8be54965d56f7"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.781159 28661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:19.808367 28661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:19.810307 28661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:19.810470 28661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:19.810997 28661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:19.811079 28661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:19.811129 28661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:19.811158 28661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:46:19 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337179, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:19.818454 28661 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:42607
I20260810 04:46:19.818536 28885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:42607 every 8 connection(s)
I20260810 04:46:19.818812 28661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:19.821285 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.811545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:56195 (local address 127.14.237.62:42627)
0810 04:46:19.811588 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:19.811591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:19.811851 (+ 260us) server_negotiation.cc:407] Connection header received
0810 04:46:19.812423 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:19.812424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:19.812438 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:19.812519 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:19.813534 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.814106 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.815229 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.815454 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.815533 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.818270 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:19.818272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:19.818272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:19.818281 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:19.819854 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.820260 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.820261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.820261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.820323 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.820595 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.820596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.820596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.820723 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:19.820796 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:19.821073 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 04:46:19.821158 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:46:19.821483 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.811541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:42627 (local address 127.14.237.1:56195)
0810 04:46:19.811721 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:19.811758 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:19.812308 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:19.812643 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:19.812648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:19.813080 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:19.813420 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:19.813428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.814212 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.814213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:19.815128 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:19.815131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.815555 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.815556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:19.815675 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.816148 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:19.816157 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:19.818168 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:19.819939 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:19.819946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:19.819948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:19.820175 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:19.820401 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:19.820402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:19.820402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:19.820516 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:19.820801 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:19.820808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:19.820943 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 04:46:19.821050 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:46:19.821207 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260810 04:46:19.822124 28886 heartbeater.cc:347] Connected to a master server at 127.14.237.62:42627
I20260810 04:46:19.822229 28886 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:19.822454 28886 heartbeater.cc:511] Master 127.14.237.62:42627 requested a full tablet report, sending...
I20260810 04:46:19.822962 28521 ts_manager.cc:194] Registered new tserver with Master: e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607)
I20260810 04:46:19.823885 28521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:56195
I20260810 04:46:19.827919 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 28661
I20260810 04:46:19.828030 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:19.828248 15284 external_mini_cluster.cc:1428] Setting key a4b47cf1f1dc5b9589109dea7f6d6838
I20260810 04:46:19.828296 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.823147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:43683 (local address 127.14.237.62:42627)
0810 04:46:19.823183 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:19.823186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:19.823321 (+ 135us) server_negotiation.cc:407] Connection header received
0810 04:46:19.823374 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:19.823375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:19.823383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:19.823427 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:19.823760 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.824280 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.825093 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.825343 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.825391 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.826109 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:19.826110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:19.826111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:19.826117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:19.827206 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.827523 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.827523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.827523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.827573 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.827746 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.827747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.827747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.827881 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:19.827955 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:19.828148 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 04:46:19.828218 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:46:19.828318 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.823113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:42627 (local address 127.14.237.1:43683)
0810 04:46:19.823238 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:19.823249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:19.823319 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:19.823433 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:19.823434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:19.823646 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:19.823700 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:19.823701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.824356 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.824357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:19.825025 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:19.825027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.825393 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.825394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:19.825437 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.825784 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:19.825788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:19.826039 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:19.827296 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:19.827297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:19.827297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:19.827458 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:19.827626 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:19.827626 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:19.827626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:19.827667 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:19.827951 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:19.827952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:19.828009 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 04:46:19.828109 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 04:46:19.828208 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab.
Entry for principal oryx/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab.
I20260810 04:46:19.854117 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:42627
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab
--principal=oryx/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:19.936614 28901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:19.936800 28901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:19.936820 28901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:19.937893 28901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:19.937928 28901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:19.937948 28901 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:46:19.937964 28901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:19.939325 28901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab
--principal=oryx/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:42627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:19.939536 28901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:19.939689 28901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:19.944747 28907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:19.944785 28914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:19.944764 28910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:19.944969 28901 server_base.cc:1034] running on GCE node
I20260810 04:46:19.945182 28901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:19.945488 28901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:19.946641 28901 hybrid_clock.cc:648] HybridClock initialized: now 1786337179946631 us; error 38 us; skew 500 ppm
Aug 10 04:46:19 dist-test-slave-8hhm krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337179, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:19.949159 28901 init.cc:377] Logged in from keytab as oryx/127.14.237.2@KRBTEST.COM (short username oryx)
I20260810 04:46:19.949659 28901 webserver.cc:460] Webserver started at http://127.14.237.2:42089/ using document root <none> and password file <none>
I20260810 04:46:19.949856 28901 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:19.949896 28901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:19.949982 28901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:19.950906 28901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "c21ea80db8bb4c9692a1ee2abdcdc485"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "35e3ec9b9d18291c25a10f53b20c0e22"
server_key_iv: "3ba73f692310cc22d37da8f2c122e2b8"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.951192 28901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "c21ea80db8bb4c9692a1ee2abdcdc485"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "35e3ec9b9d18291c25a10f53b20c0e22"
server_key_iv: "3ba73f692310cc22d37da8f2c122e2b8"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.952423 28901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:19.953070 28927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:19.953248 28901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:19.953336 28901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "c21ea80db8bb4c9692a1ee2abdcdc485"
format_stamp: "Formatted at 2026-08-10 04:46:19 on dist-test-slave-8hhm"
server_key: "35e3ec9b9d18291c25a10f53b20c0e22"
server_key_iv: "3ba73f692310cc22d37da8f2c122e2b8"
server_key_version: "encryptionkey@0"
I20260810 04:46:19.953389 28901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:19.967317 28901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:19.969118 28901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:19.969317 28901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:19.969798 28901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:19.969854 28901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:19.969887 28901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:19.969913 28901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:19.975980 28901 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:36575
I20260810 04:46:19.976070 29136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:36575 every 8 connection(s)
I20260810 04:46:19.976375 28901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:46:19 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337179, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:19.979252 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 28901
I20260810 04:46:19.979385 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:19.979650 15284 external_mini_cluster.cc:1428] Setting key 1fc9c6b1b73203360f8b257998262408
I20260810 04:46:19.980369 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.970411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:35301 (local address 127.14.237.62:42627)
0810 04:46:19.970462 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:19.970464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:19.970659 (+ 195us) server_negotiation.cc:407] Connection header received
0810 04:46:19.971307 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:19.971309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:19.971323 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:19.971358 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:19.972383 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.973262 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.974641 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.974988 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.975063 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.977662 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:19.977664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:19.977664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:19.977671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:19.978845 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.979253 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.979254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.979255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.979331 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.979635 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.979636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.979636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.979781 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:19.979859 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:19.980199 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 04:46:19.980280 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:46:19.980544 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.970324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:42627 (local address 127.14.237.2:35301)
0810 04:46:19.970523 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:19.970561 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:19.971167 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:19.971547 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:19.971552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:19.971967 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:19.972252 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:19.972259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.973353 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.973354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:19.974523 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:19.974526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.975090 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.975092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:19.975304 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.975868 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:19.975878 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:19.977519 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:19.978924 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:19.978929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:19.978931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:19.979148 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:19.979433 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:19.979434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:19.979434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:19.979561 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:19.979887 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:19.979893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:19.980062 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 04:46:19.980176 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 04:46:19.980332 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260810 04:46:19.981573 29137 heartbeater.cc:347] Connected to a master server at 127.14.237.62:42627
I20260810 04:46:19.981668 29137 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:19.981860 29137 heartbeater.cc:511] Master 127.14.237.62:42627 requested a full tablet report, sending...
I20260810 04:46:19.982223 28521 ts_manager.cc:194] Registered new tserver with Master: c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575)
I20260810 04:46:19.982692 28521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:35301
WARNING: no policy specified for oryx/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.3@KRBTEST.COM" created.
I20260810 04:46:19.994813 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.982643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:57773 (local address 127.14.237.62:42627)
0810 04:46:19.982698 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:19.982700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:19.988717 (+ 6017us) server_negotiation.cc:407] Connection header received
0810 04:46:19.988744 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:19.988745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:19.988769 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:19.988820 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:19.989108 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.989680 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.990428 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.990613 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.990656 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.991619 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:19.991621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:19.991621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:19.991628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:19.993538 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.993895 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.993896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.993896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.993967 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:19.994221 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:19.994221 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:19.994222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:19.994365 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:19.994427 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:19.994659 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 04:46:19.994731 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:46:19.994874 29151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:19.982572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:42627 (local address 127.14.237.2:57773)
0810 04:46:19.988614 (+ 6042us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:19.988624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:19.988680 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:19.988837 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:19.988838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:19.988991 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:19.989052 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:19.989053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.989772 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.989773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:19.990295 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:19.990297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:19.990672 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:19.990672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:19.990721 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:19.991222 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:19.991226 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:19.991513 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:19.993643 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:19.993644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:19.993645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:19.993818 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:19.994072 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:19.994073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:19.994073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:19.994149 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:19.994439 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:19.994440 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:19.994494 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 04:46:19.994575 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:46:19.994700 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6014,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab.
Entry for principal oryx/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab.
I20260810 04:46:20.008172 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:42627
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab
--principal=oryx/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:20.091334 29155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:20.091547 29155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:20.091575 29155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:20.092634 29155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:20.092667 29155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:20.092686 29155 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:46:20.092703 29155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:20.094262 29155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab
--principal=oryx/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.237.62:42627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:20.094441 29155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:20.094620 29155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:20.099761 29169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:20.099749 29165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:20.099769 29166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:20.100047 29155 server_base.cc:1034] running on GCE node
I20260810 04:46:20.100296 29155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:20.100545 29155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:20.101691 29155 hybrid_clock.cc:648] HybridClock initialized: now 1786337180101681 us; error 31 us; skew 500 ppm
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337180, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:20.104293 29155 init.cc:377] Logged in from keytab as oryx/127.14.237.3@KRBTEST.COM (short username oryx)
I20260810 04:46:20.104667 29155 webserver.cc:460] Webserver started at http://127.14.237.3:40873/ using document root <none> and password file <none>
I20260810 04:46:20.104853 29155 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:20.104892 29155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:20.104979 29155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:20.106014 29155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "03388ea92e124181a960c753236da9f1"
format_stamp: "Formatted at 2026-08-10 04:46:20 on dist-test-slave-8hhm"
server_key: "5c47c3d20aa6362b34db307c31bf221a"
server_key_iv: "57bb12ec4849d63217fee3000494ee68"
server_key_version: "encryptionkey@0"
I20260810 04:46:20.106329 29155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "03388ea92e124181a960c753236da9f1"
format_stamp: "Formatted at 2026-08-10 04:46:20 on dist-test-slave-8hhm"
server_key: "5c47c3d20aa6362b34db307c31bf221a"
server_key_iv: "57bb12ec4849d63217fee3000494ee68"
server_key_version: "encryptionkey@0"
I20260810 04:46:20.107553 29155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:46:20.108289 29178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.108460 29155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:20.108532 29155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "03388ea92e124181a960c753236da9f1"
format_stamp: "Formatted at 2026-08-10 04:46:20 on dist-test-slave-8hhm"
server_key: "5c47c3d20aa6362b34db307c31bf221a"
server_key_iv: "57bb12ec4849d63217fee3000494ee68"
server_key_version: "encryptionkey@0"
I20260810 04:46:20.108584 29155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:20.136808 29155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:20.139240 29155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:20.139346 29155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:20.139847 29155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:20.139906 29155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.139943 29155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:20.139982 29155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.146852 29155 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:36347
I20260810 04:46:20.146941 29387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:36347 every 8 connection(s)
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337180, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:20.147301 29155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:20.150336 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.140359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:35325 (local address 127.14.237.62:42627)
0810 04:46:20.140423 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.140425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.140635 (+ 210us) server_negotiation.cc:407] Connection header received
0810 04:46:20.141326 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.141328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.141342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.141390 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:20.142339 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.143223 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.144851 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.145080 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.145161 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.147751 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:20.147752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:20.147753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:20.147761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:20.148990 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.149462 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.149463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.149463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.149518 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.149781 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.149781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.149781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.149918 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:20.149995 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.150206 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.150273 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:46:20.150497 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.140338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:42627 (local address 127.14.237.3:35325)
0810 04:46:20.140513 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.140549 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.141204 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.141484 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.141489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.141925 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:20.142220 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.142226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.143345 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.143346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.144758 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.144762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.145168 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.145169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.145331 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.145926 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:20.145938 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:20.147653 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:20.149060 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:20.149066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:20.149068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:20.149403 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:20.149614 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:20.149615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:20.149615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:20.149733 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:20.149991 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:20.149995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:20.150107 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.150185 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.150309 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260810 04:46:20.151028 29389 heartbeater.cc:347] Connected to a master server at 127.14.237.62:42627
I20260810 04:46:20.151131 29389 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:20.151314 29389 heartbeater.cc:511] Master 127.14.237.62:42627 requested a full tablet report, sending...
I20260810 04:46:20.151660 28521 ts_manager.cc:194] Registered new tserver with Master: 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347)
I20260810 04:46:20.152069 28521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:35325
I20260810 04:46:20.153944 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 29155
I20260810 04:46:20.154059 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:20.154266 15284 external_mini_cluster.cc:1428] Setting key 766de9f8208c1c011ef11a561b950830
I20260810 04:46:20.155541 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:46:20.157845 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.152158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:44153 (local address 127.14.237.62:42627)
0810 04:46:20.152196 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.152198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.152340 (+ 142us) server_negotiation.cc:407] Connection header received
0810 04:46:20.152407 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.152408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.152420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.152477 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:20.152758 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.153335 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.154027 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.154293 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.154367 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.155294 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:20.155295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:20.155296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:20.155303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:20.156685 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.157043 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.157044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.157045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.157099 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.157344 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.157345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.157346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.157453 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:20.157521 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.157688 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.157768 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:46:20.157855 29396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.152103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:42627 (local address 127.14.237.3:44153)
0810 04:46:20.152262 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.152274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.152337 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.152489 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.152489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.152636 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:20.152686 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.152687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.153420 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.153421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.153928 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.153930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.154367 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.154368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.154442 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.154814 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:20.154818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:20.155200 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:20.156770 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:20.156771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:20.156771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:20.156978 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:20.157176 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:20.157177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:20.157177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:20.157261 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:20.157526 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:20.157526 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:20.157570 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.157647 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.157752 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:46:20.178225 28624 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:20.172972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.14.237.62:42627)
0810 04:46:20.173042 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.173044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.173068 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:46:20.173137 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.173137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.173147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.173201 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:20.173789 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.174251 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.174971 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.175129 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.175172 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.178169 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35176: BlockingRecv error: recv got EOF from 127.0.0.1:35176 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:20.185621 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.179111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.14.237.62:42627)
0810 04:46:20.179170 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.179171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.179243 (+ 72us) server_negotiation.cc:407] Connection header received
0810 04:46:20.179297 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.179297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.179306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.179368 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:20.179780 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.180204 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.180810 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.181032 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.181085 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.183297 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:20.183298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:20.183299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:20.183305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:20.184629 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.184941 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.184942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.184942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.184994 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.185158 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.185159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.185159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.185337 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:20.185416 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.185505 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.185560 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:46:20.187080 28521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:46:20.188643 28521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:20.195115 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.191039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42607 (local address 127.0.0.1:32958)
0810 04:46:20.191297 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.191307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.191361 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.191647 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.191650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.191652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.191716 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.191717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.192762 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.192763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.193916 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.193918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.194808 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.194809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.194901 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.194915 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.194986 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.195038 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260810 04:46:20.195140 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.191297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.14.237.1:42607)
0810 04:46:20.191481 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.191486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.191495 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:20.191536 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.191537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.191549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.191643 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.191831 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.192677 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.194014 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.194711 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.194794 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.194884 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.195001 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.195036 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260810 04:46:20.195503 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.190957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:36347 (local address 127.0.0.1:43480)
0810 04:46:20.191239 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.191286 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.191354 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.191817 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.191817 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.191820 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.191871 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.191873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.193153 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.193154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.194152 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.194154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.195110 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.195111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.195318 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.195323 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.195383 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.195434 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":48,"threads_started":1}
I20260810 04:46:20.195535 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.191044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.14.237.3:36347)
0810 04:46:20.191630 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.191635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.191645 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:20.191682 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.191684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.191698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.191804 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.191963 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.193055 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.194251 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.194937 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.195007 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.195073 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.195402 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.195446 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":135,"threads_started":1}
I20260810 04:46:20.195659 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.191040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:36575 (local address 127.0.0.1:58788)
0810 04:46:20.191463 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.191473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.191510 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.192136 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.192137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.192139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.192183 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.192184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.193326 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.193327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.194407 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.194410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.195450 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.195451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.195505 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.195509 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.195540 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.195578 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":116,"threads_started":1}
I20260810 04:46:20.196000 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.191517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.14.237.2:36575)
0810 04:46:20.191933 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.191937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.191946 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:20.191983 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.191986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.192003 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.192104 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.192266 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.193261 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.194492 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.195365 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.195717 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.195815 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.195844 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.195892 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":83,"threads_started":1}
I20260810 04:46:20.195889 28764 tablet_service.cc:1467] Processing CreateTablet for tablet 75de6404a30842c3bfcfc9ba2c46296e (DEFAULT_TABLE table=test-table [id=d36fa2b92cf04b799628b12b91b76bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.196291 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75de6404a30842c3bfcfc9ba2c46296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.196461 29264 tablet_service.cc:1467] Processing CreateTablet for tablet 75de6404a30842c3bfcfc9ba2c46296e (DEFAULT_TABLE table=test-table [id=d36fa2b92cf04b799628b12b91b76bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.196800 29264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75de6404a30842c3bfcfc9ba2c46296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.197636 29005 tablet_service.cc:1467] Processing CreateTablet for tablet 75de6404a30842c3bfcfc9ba2c46296e (DEFAULT_TABLE table=test-table [id=d36fa2b92cf04b799628b12b91b76bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.197927 29005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75de6404a30842c3bfcfc9ba2c46296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.200474 29428 tablet_bootstrap.cc:492] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Bootstrap starting.
I20260810 04:46:20.200949 29430 tablet_bootstrap.cc:492] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Bootstrap starting.
I20260810 04:46:20.201202 29428 tablet_bootstrap.cc:654] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.201385 29429 tablet_bootstrap.cc:492] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: Bootstrap starting.
I20260810 04:46:20.201589 29428 log.cc:824] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:20.201679 29430 tablet_bootstrap.cc:654] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.202001 29429 tablet_bootstrap.cc:654] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.202015 29430 log.cc:824] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:20.202347 29429 log.cc:824] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:20.202382 29428 tablet_bootstrap.cc:492] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: No bootstrap required, opened a new log
I20260810 04:46:20.202462 29428 ts_tablet_manager.cc:1397] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:20.202936 29430 tablet_bootstrap.cc:492] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: No bootstrap required, opened a new log
I20260810 04:46:20.203055 29429 tablet_bootstrap.cc:492] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: No bootstrap required, opened a new log
I20260810 04:46:20.203068 29430 ts_tablet_manager.cc:1397] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:46:20.203140 29429 ts_tablet_manager.cc:1397] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:20.204521 29428 raft_consensus.cc:348] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.204706 29428 raft_consensus.cc:374] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.204753 29428 raft_consensus.cc:729] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e243ef1700264f1f986d1875c61e5fef, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.204871 29428 consensus_queue.cc:260] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.204821 29429 raft_consensus.cc:348] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.205008 29429 raft_consensus.cc:374] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.205042 29429 raft_consensus.cc:729] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03388ea92e124181a960c753236da9f1, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.205139 29428 ts_tablet_manager.cc:1428] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:46:20.205065 29430 raft_consensus.cc:348] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.205159 29429 consensus_queue.cc:260] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.205245 29430 raft_consensus.cc:374] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.205286 28886 heartbeater.cc:503] Master 127.14.237.62:42627 was elected leader, sending a full tablet report...
I20260810 04:46:20.205287 29430 raft_consensus.cc:729] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21ea80db8bb4c9692a1ee2abdcdc485, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.205413 29430 consensus_queue.cc:260] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.205452 29429 ts_tablet_manager.cc:1428] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:46:20.205606 29389 heartbeater.cc:503] Master 127.14.237.62:42627 was elected leader, sending a full tablet report...
I20260810 04:46:20.205996 29137 heartbeater.cc:503] Master 127.14.237.62:42627 was elected leader, sending a full tablet report...
I20260810 04:46:20.206360 29430 ts_tablet_manager.cc:1428] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260810 04:46:20.226970 29140 tablet.cc:2369] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:46:20.319682 28887 tablet.cc:2369] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:46:20.348336 29436 raft_consensus.cc:482] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:20.348469 29436 raft_consensus.cc:504] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.348922 29436 leader_election.cc:288] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347), c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575)
I20260810 04:46:20.353036 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.349356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:48989 (local address 127.14.237.3:36347)
0810 04:46:20.349398 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.349400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.349406 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:46:20.349439 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.349440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.349451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.349523 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.349711 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.350482 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.352092 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.352681 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.352764 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.352774 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.352920 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.352969 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:46:20.353034 29441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.349123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:36347 (local address 127.14.237.1:48989)
0810 04:46:20.349309 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.349321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.349385 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.349542 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.349543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.349545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.349628 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.349631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.350555 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.350555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.351991 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.351995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.352782 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.352783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.352838 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.352842 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.352889 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.352926 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":67,"thread_start_us":72,"threads_started":1}
I20260810 04:46:20.353123 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.349272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:36189 (local address 127.14.237.2:36575)
0810 04:46:20.349335 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.349337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.349379 (+ 42us) server_negotiation.cc:407] Connection header received
0810 04:46:20.349439 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.349440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.349451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.349524 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.349710 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.350461 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.352104 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.352709 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.352770 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.352777 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.353003 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.353055 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:46:20.353153 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.349123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:36575 (local address 127.14.237.1:36189)
0810 04:46:20.349296 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.349321 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.349385 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.349542 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.349543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.349547 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.349628 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.349631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.350554 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.350555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.351991 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.351995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.352819 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.352821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.352919 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.352924 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.352978 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.353023 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":98,"spinlock_wait_cycles":16640,"thread_start_us":99,"threads_started":1}
I20260810 04:46:20.353471 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75de6404a30842c3bfcfc9ba2c46296e" candidate_uuid: "e243ef1700264f1f986d1875c61e5fef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03388ea92e124181a960c753236da9f1" is_pre_election: true
I20260810 04:46:20.353508 29050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75de6404a30842c3bfcfc9ba2c46296e" candidate_uuid: "e243ef1700264f1f986d1875c61e5fef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" is_pre_election: true
I20260810 04:46:20.353646 29310 raft_consensus.cc:2454] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e243ef1700264f1f986d1875c61e5fef in term 0.
I20260810 04:46:20.353657 29050 raft_consensus.cc:2454] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e243ef1700264f1f986d1875c61e5fef in term 0.
I20260810 04:46:20.353909 28692 leader_election.cc:302] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c21ea80db8bb4c9692a1ee2abdcdc485, e243ef1700264f1f986d1875c61e5fef; no voters:
I20260810 04:46:20.354050 29436 raft_consensus.cc:2781] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:20.354115 29436 raft_consensus.cc:482] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:20.354138 29436 raft_consensus.cc:3037] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.354820 29436 raft_consensus.cc:504] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.354976 29436 leader_election.cc:288] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [CANDIDATE]: Term 1 election: Requested vote from peers 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347), c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575)
I20260810 04:46:20.355249 29050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75de6404a30842c3bfcfc9ba2c46296e" candidate_uuid: "e243ef1700264f1f986d1875c61e5fef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485"
I20260810 04:46:20.355264 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75de6404a30842c3bfcfc9ba2c46296e" candidate_uuid: "e243ef1700264f1f986d1875c61e5fef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03388ea92e124181a960c753236da9f1"
I20260810 04:46:20.355333 29310 raft_consensus.cc:3037] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.355333 29050 raft_consensus.cc:3037] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.356212 29050 raft_consensus.cc:2454] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e243ef1700264f1f986d1875c61e5fef in term 1.
I20260810 04:46:20.356212 29310 raft_consensus.cc:2454] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e243ef1700264f1f986d1875c61e5fef in term 1.
I20260810 04:46:20.356400 28692 leader_election.cc:302] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c21ea80db8bb4c9692a1ee2abdcdc485, e243ef1700264f1f986d1875c61e5fef; no voters:
I20260810 04:46:20.356519 29436 raft_consensus.cc:2781] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:20.356642 29436 raft_consensus.cc:686] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 1 LEADER]: Becoming Leader. State: Replica: e243ef1700264f1f986d1875c61e5fef, State: Running, Role: LEADER
I20260810 04:46:20.356840 29436 consensus_queue.cc:237] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } }
I20260810 04:46:20.357697 28521 catalog_manager.cc:5495] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef reported cstate change: term changed from 0 to 1, leader changed from <none> to e243ef1700264f1f986d1875c61e5fef (127.14.237.1). New cstate: current_term: 1 leader_uuid: "e243ef1700264f1f986d1875c61e5fef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } health_report { overall_health: UNKNOWN } } }
I20260810 04:46:20.376890 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.374688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.14.237.1:42607)
0810 04:46:20.374732 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.374734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.374744 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:20.374853 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.374853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.374877 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.374937 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:20.375043 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.375484 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.376203 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.376393 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.376449 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.376578 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:20.376676 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:20.376733 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.376766 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.376782 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:46:20.378914 29050 raft_consensus.cc:1261] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Refusing update from remote peer e243ef1700264f1f986d1875c61e5fef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:20.378914 29310 raft_consensus.cc:1261] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Refusing update from remote peer e243ef1700264f1f986d1875c61e5fef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:20.379272 29444 consensus_queue.cc:1035] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [LEADER]: Connected to new peer: Peer: permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:20.379417 29436 consensus_queue.cc:1035] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:20.384183 28522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:20.384290 28522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:20.385129 28522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: Sending DeleteTablet for 3 replicas of tablet 75de6404a30842c3bfcfc9ba2c46296e
I20260810 04:46:20.385560 28764 tablet_service.cc:1514] Processing DeleteTablet for tablet 75de6404a30842c3bfcfc9ba2c46296e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:20 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:32958
I20260810 04:46:20.385571 29005 tablet_service.cc:1514] Processing DeleteTablet for tablet 75de6404a30842c3bfcfc9ba2c46296e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:20 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:58788
I20260810 04:46:20.385833 29264 tablet_service.cc:1514] Processing DeleteTablet for tablet 75de6404a30842c3bfcfc9ba2c46296e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:20 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:43480
I20260810 04:46:20.385952 28522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 04:46:20.386037 29465 tablet_replica.cc:330] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: stopping tablet replica
W20260810 04:46:20.386129 28522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:20.386154 29465 raft_consensus.cc:2229] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:20.386261 29465 raft_consensus.cc:2258] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:20.386443 29468 tablet_replica.cc:330] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: stopping tablet replica
I20260810 04:46:20.386529 29468 raft_consensus.cc:2229] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:20.386622 29468 raft_consensus.cc:2258] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:20.387002 29465 ts_tablet_manager.cc:1905] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:20.387441 28764 tablet_service.cc:1467] Processing CreateTablet for tablet 9b13c8caa915454d9bb6b5e2594c8ae1 (DEFAULT_TABLE table=test-table [id=6af81cad837e46c1a32c3c82a1bf1226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.387575 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b13c8caa915454d9bb6b5e2594c8ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.387557 29005 tablet_service.cc:1467] Processing CreateTablet for tablet 9b13c8caa915454d9bb6b5e2594c8ae1 (DEFAULT_TABLE table=test-table [id=6af81cad837e46c1a32c3c82a1bf1226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.387650 29005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b13c8caa915454d9bb6b5e2594c8ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.388726 29465 ts_tablet_manager.cc:1918] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:20.388790 29465 log.cc:1196] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/75de6404a30842c3bfcfc9ba2c46296e
I20260810 04:46:20.388870 29430 tablet_bootstrap.cc:492] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: Bootstrap starting.
I20260810 04:46:20.388871 29428 tablet_bootstrap.cc:492] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: Bootstrap starting.
I20260810 04:46:20.389320 29428 tablet_bootstrap.cc:654] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.389320 29430 tablet_bootstrap.cc:654] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.389369 29264 tablet_service.cc:1467] Processing CreateTablet for tablet 9b13c8caa915454d9bb6b5e2594c8ae1 (DEFAULT_TABLE table=test-table [id=6af81cad837e46c1a32c3c82a1bf1226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.389495 29264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b13c8caa915454d9bb6b5e2594c8ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.389873 29465 ts_tablet_manager.cc:1939] T 75de6404a30842c3bfcfc9ba2c46296e P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting consensus metadata
I20260810 04:46:20.389923 29430 tablet_bootstrap.cc:492] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: No bootstrap required, opened a new log
I20260810 04:46:20.389964 29430 ts_tablet_manager.cc:1397] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:20.390100 29430 raft_consensus.cc:348] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.390165 29430 raft_consensus.cc:374] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.390185 29430 raft_consensus.cc:729] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21ea80db8bb4c9692a1ee2abdcdc485, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.390234 29430 consensus_queue.cc:260] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.390349 29430 ts_tablet_manager.cc:1428] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.390909 29428 tablet_bootstrap.cc:492] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: No bootstrap required, opened a new log
I20260810 04:46:20.390964 29428 ts_tablet_manager.cc:1397] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:46:20.390985 29429 tablet_bootstrap.cc:492] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: Bootstrap starting.
I20260810 04:46:20.391105 29428 raft_consensus.cc:348] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.391161 29428 raft_consensus.cc:374] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.391175 29428 raft_consensus.cc:729] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e243ef1700264f1f986d1875c61e5fef, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.391155 28497 catalog_manager.cc:4841] TS c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575): tablet 75de6404a30842c3bfcfc9ba2c46296e (table test-table [id=d36fa2b92cf04b799628b12b91b76bfd]) successfully deleted
I20260810 04:46:20.391239 29428 consensus_queue.cc:260] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.391323 29428 ts_tablet_manager.cc:1428] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.391613 29468 ts_tablet_manager.cc:1905] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:20.391709 29429 tablet_bootstrap.cc:654] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.392422 29429 tablet_bootstrap.cc:492] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: No bootstrap required, opened a new log
I20260810 04:46:20.392480 29429 ts_tablet_manager.cc:1397] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:20.392630 29429 raft_consensus.cc:348] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.392704 29429 raft_consensus.cc:374] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.392724 29429 raft_consensus.cc:729] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03388ea92e124181a960c753236da9f1, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.392766 29429 consensus_queue.cc:260] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.392854 29429 ts_tablet_manager.cc:1428] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.393321 29468 ts_tablet_manager.cc:1918] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:20.393414 29468 log.cc:1196] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/75de6404a30842c3bfcfc9ba2c46296e
I20260810 04:46:20.393683 29468 ts_tablet_manager.cc:1939] T 75de6404a30842c3bfcfc9ba2c46296e P e243ef1700264f1f986d1875c61e5fef: Deleting consensus metadata
I20260810 04:46:20.394089 29469 tablet_replica.cc:330] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: stopping tablet replica
I20260810 04:46:20.394107 28492 catalog_manager.cc:4841] TS e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607): tablet 75de6404a30842c3bfcfc9ba2c46296e (table test-table [id=d36fa2b92cf04b799628b12b91b76bfd]) successfully deleted
I20260810 04:46:20.394208 29469 raft_consensus.cc:2229] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:20.394335 29469 raft_consensus.cc:2258] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:20.394727 29469 ts_tablet_manager.cc:1905] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:20.396008 29469 ts_tablet_manager.cc:1918] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:20.396097 29469 log.cc:1196] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/75de6404a30842c3bfcfc9ba2c46296e
I20260810 04:46:20.396324 29469 ts_tablet_manager.cc:1939] T 75de6404a30842c3bfcfc9ba2c46296e P 03388ea92e124181a960c753236da9f1: Deleting consensus metadata
I20260810 04:46:20.396703 28495 catalog_manager.cc:4841] TS 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347): tablet 75de6404a30842c3bfcfc9ba2c46296e (table test-table [id=d36fa2b92cf04b799628b12b91b76bfd]) successfully deleted
W20260810 04:46:20.397790 29391 tablet.cc:2369] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:46:20.511018 29439 raft_consensus.cc:482] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:20.511111 29439 raft_consensus.cc:504] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.511464 29439 leader_election.cc:288] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607), c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575)
I20260810 04:46:20.514297 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.511725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:39793 (local address 127.14.237.2:36575)
0810 04:46:20.511790 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.511806 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.511819 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:46:20.511849 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.511850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.511860 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.511911 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.512079 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.512559 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.513580 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.513951 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.514037 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.514047 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.514193 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.514229 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:46:20.514297 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.511627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:36575 (local address 127.14.237.3:39793)
0810 04:46:20.511691 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.511708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.511776 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.511924 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.511925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.511927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.512001 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.512019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.512633 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.512633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.513453 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.513456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.514035 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.514036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.514113 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.514115 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.514187 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.514227 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 04:46:20.514575 29050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b13c8caa915454d9bb6b5e2594c8ae1" candidate_uuid: "03388ea92e124181a960c753236da9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" is_pre_election: true
I20260810 04:46:20.514681 29050 raft_consensus.cc:2454] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03388ea92e124181a960c753236da9f1 in term 0.
I20260810 04:46:20.514897 29185 leader_election.cc:302] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03388ea92e124181a960c753236da9f1, c21ea80db8bb4c9692a1ee2abdcdc485; no voters:
I20260810 04:46:20.515027 29439 raft_consensus.cc:2781] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:20.515069 29439 raft_consensus.cc:482] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:20.515107 29439 raft_consensus.cc:3037] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.515177 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.511774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:35687 (local address 127.14.237.1:42607)
0810 04:46:20.511846 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.511848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.511935 (+ 87us) server_negotiation.cc:407] Connection header received
0810 04:46:20.512008 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.512009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.512018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.512074 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.512809 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.513336 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.514259 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.514815 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.514893 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.514900 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.515076 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.515116 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:46:20.515192 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.511640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42607 (local address 127.14.237.3:35687)
0810 04:46:20.511833 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.511844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.511942 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.512645 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.512646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.512648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.512720 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.512730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.513430 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.513431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.514162 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.514163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.514893 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.514894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.514962 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.514965 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.515030 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.515071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":62,"threads_started":1}
I20260810 04:46:20.515403 28809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b13c8caa915454d9bb6b5e2594c8ae1" candidate_uuid: "03388ea92e124181a960c753236da9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e243ef1700264f1f986d1875c61e5fef" is_pre_election: true
I20260810 04:46:20.515528 28809 raft_consensus.cc:2454] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03388ea92e124181a960c753236da9f1 in term 0.
I20260810 04:46:20.516001 29439 raft_consensus.cc:504] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.516156 29439 leader_election.cc:288] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [CANDIDATE]: Term 1 election: Requested vote from peers e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607), c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575)
I20260810 04:46:20.516289 28809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b13c8caa915454d9bb6b5e2594c8ae1" candidate_uuid: "03388ea92e124181a960c753236da9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e243ef1700264f1f986d1875c61e5fef"
I20260810 04:46:20.516299 29050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b13c8caa915454d9bb6b5e2594c8ae1" candidate_uuid: "03388ea92e124181a960c753236da9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485"
I20260810 04:46:20.516367 29050 raft_consensus.cc:3037] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.516383 28809 raft_consensus.cc:3037] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.517031 28809 raft_consensus.cc:2454] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03388ea92e124181a960c753236da9f1 in term 1.
I20260810 04:46:20.517042 29050 raft_consensus.cc:2454] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03388ea92e124181a960c753236da9f1 in term 1.
I20260810 04:46:20.517197 29185 leader_election.cc:302] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03388ea92e124181a960c753236da9f1, c21ea80db8bb4c9692a1ee2abdcdc485; no voters:
I20260810 04:46:20.517326 29439 raft_consensus.cc:2781] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:20.517444 29439 raft_consensus.cc:686] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 1 LEADER]: Becoming Leader. State: Replica: 03388ea92e124181a960c753236da9f1, State: Running, Role: LEADER
I20260810 04:46:20.517510 29439 consensus_queue.cc:237] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.518246 28522 catalog_manager.cc:5495] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03388ea92e124181a960c753236da9f1 (127.14.237.3). New cstate: current_term: 1 leader_uuid: "03388ea92e124181a960c753236da9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } health_report { overall_health: HEALTHY } } }
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337179, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.62@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:20.561098 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.554031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.14.237.62:42627)
0810 04:46:20.554119 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.554123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.554248 (+ 125us) server_negotiation.cc:407] Connection header received
0810 04:46:20.554344 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.554345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.554356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.554425 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:20.554914 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.555460 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.556164 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.556354 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.556431 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.558618 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:20.558619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:20.558620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:20.558627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:20.559848 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.560228 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.560230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.560231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.560289 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.560487 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.560488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.560488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.560598 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:20.560670 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.560840 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.561001 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":56}
I20260810 04:46:20.561098 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.553944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:42627 (local address 127.0.0.1:35194)
0810 04:46:20.554161 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.554170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.554270 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.554431 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.554432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.554770 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:20.554815 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.554818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.555553 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.555554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.556079 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.556081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.556436 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.556438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.556499 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.557071 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:20.557075 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:20.558523 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:20.559943 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:20.559952 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:20.559954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:20.560151 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:20.560356 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:20.560357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:20.560357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:20.560421 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:20.560672 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:20.560681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:20.560727 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.560805 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.560946 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":131,"threads_started":1}
I20260810 04:46:20.563103 28522 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:35194:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:46:20.563324 28522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:20.564884 28764 tablet_service.cc:1467] Processing CreateTablet for tablet f7a4e4ae90ac40fa88bcd1a19b505185 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0a29a5d6984490bb47674e587dfde33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:46:20.564883 29005 tablet_service.cc:1467] Processing CreateTablet for tablet f7a4e4ae90ac40fa88bcd1a19b505185 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0a29a5d6984490bb47674e587dfde33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:46:20.564908 29264 tablet_service.cc:1467] Processing CreateTablet for tablet f7a4e4ae90ac40fa88bcd1a19b505185 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0a29a5d6984490bb47674e587dfde33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:46:20.565068 29005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a4e4ae90ac40fa88bcd1a19b505185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.565068 29264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a4e4ae90ac40fa88bcd1a19b505185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.565070 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a4e4ae90ac40fa88bcd1a19b505185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.566316 29428 tablet_bootstrap.cc:492] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: Bootstrap starting.
I20260810 04:46:20.566573 29430 tablet_bootstrap.cc:492] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: Bootstrap starting.
I20260810 04:46:20.566697 29428 tablet_bootstrap.cc:654] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.567003 29430 tablet_bootstrap.cc:654] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.567379 29428 tablet_bootstrap.cc:492] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: No bootstrap required, opened a new log
I20260810 04:46:20.567380 29429 tablet_bootstrap.cc:492] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: Bootstrap starting.
I20260810 04:46:20.567451 29428 ts_tablet_manager.cc:1397] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:20.567601 29428 raft_consensus.cc:348] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.567703 29428 raft_consensus.cc:374] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.567736 29428 raft_consensus.cc:729] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e243ef1700264f1f986d1875c61e5fef, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.567804 29429 tablet_bootstrap.cc:654] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.567790 29428 consensus_queue.cc:260] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.567911 29428 ts_tablet_manager.cc:1428] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.567965 29436 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.568069 29436 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:46:20.568449 29430 tablet_bootstrap.cc:492] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: No bootstrap required, opened a new log
I20260810 04:46:20.568517 29430 ts_tablet_manager.cc:1397] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:46:20.568694 29430 raft_consensus.cc:348] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.568785 29430 raft_consensus.cc:374] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.568816 29430 raft_consensus.cc:729] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21ea80db8bb4c9692a1ee2abdcdc485, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.568881 29430 consensus_queue.cc:260] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.568981 29430 ts_tablet_manager.cc:1428] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.569078 29440 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.569197 29429 tablet_bootstrap.cc:492] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: No bootstrap required, opened a new log
I20260810 04:46:20.569217 29440 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:46:20.569288 29429 ts_tablet_manager.cc:1397] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:20.569432 29429 raft_consensus.cc:348] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.569515 29429 raft_consensus.cc:374] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.569545 29429 raft_consensus.cc:729] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03388ea92e124181a960c753236da9f1, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.569597 29429 consensus_queue.cc:260] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.569708 29429 ts_tablet_manager.cc:1428] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.569761 29439 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.569870 29439 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:46:20.707365 29440 raft_consensus.cc:482] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:20.707490 29440 raft_consensus.cc:504] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.707940 29440 leader_election.cc:288] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607), 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347)
I20260810 04:46:20.711326 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.708217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:56773 (local address 127.14.237.3:36347)
0810 04:46:20.708275 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.708278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.708372 (+ 94us) server_negotiation.cc:407] Connection header received
0810 04:46:20.708450 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.708451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.708469 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.708535 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.708690 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.709198 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.710458 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.710966 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.711055 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.711063 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.711197 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.711253 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 04:46:20.711328 29492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.708127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:36347 (local address 127.14.237.2:56773)
0810 04:46:20.708293 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.708303 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.708390 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.708539 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.708540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.708542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.708607 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.708610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.709334 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.709335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.710270 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.710273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.711062 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.711063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.711124 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.711128 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.711175 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.711203 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":47,"thread_start_us":101,"threads_started":1}
I20260810 04:46:20.711422 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.708263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:56117 (local address 127.14.237.1:42607)
0810 04:46:20.708308 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.708310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.708410 (+ 100us) server_negotiation.cc:407] Connection header received
0810 04:46:20.708502 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.708503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.708512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.708594 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:20.708759 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.709316 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.710411 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.710964 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.711018 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.711030 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.711295 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.711348 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:46:20.711630 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.708134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42607 (local address 127.14.237.2:56117)
0810 04:46:20.708333 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.708348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.708427 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.708600 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.708600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.708602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:20.708672 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.708673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.709410 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.709412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.710267 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.710269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.711092 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.711093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.711180 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.711185 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.711282 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.711320 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260810 04:46:20.711762 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a4e4ae90ac40fa88bcd1a19b505185" candidate_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03388ea92e124181a960c753236da9f1" is_pre_election: true
I20260810 04:46:20.711890 29310 raft_consensus.cc:2454] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c21ea80db8bb4c9692a1ee2abdcdc485 in term 0.
I20260810 04:46:20.711905 28809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a4e4ae90ac40fa88bcd1a19b505185" candidate_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e243ef1700264f1f986d1875c61e5fef" is_pre_election: true
I20260810 04:46:20.712004 28809 raft_consensus.cc:2454] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c21ea80db8bb4c9692a1ee2abdcdc485 in term 0.
I20260810 04:46:20.712112 28930 leader_election.cc:302] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03388ea92e124181a960c753236da9f1, c21ea80db8bb4c9692a1ee2abdcdc485; no voters:
I20260810 04:46:20.712272 29440 raft_consensus.cc:2781] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:20.712323 29440 raft_consensus.cc:482] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:20.712352 29440 raft_consensus.cc:3037] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.713063 29440 raft_consensus.cc:504] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.713172 29440 leader_election.cc:288] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [CANDIDATE]: Term 1 election: Requested vote from peers e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607), 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347)
I20260810 04:46:20.713338 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a4e4ae90ac40fa88bcd1a19b505185" candidate_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03388ea92e124181a960c753236da9f1"
I20260810 04:46:20.713352 28809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a4e4ae90ac40fa88bcd1a19b505185" candidate_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e243ef1700264f1f986d1875c61e5fef"
I20260810 04:46:20.713418 29310 raft_consensus.cc:3037] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.713425 28809 raft_consensus.cc:3037] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:20.714040 28809 raft_consensus.cc:2454] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c21ea80db8bb4c9692a1ee2abdcdc485 in term 1.
I20260810 04:46:20.714092 29310 raft_consensus.cc:2454] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c21ea80db8bb4c9692a1ee2abdcdc485 in term 1.
I20260810 04:46:20.714273 28929 leader_election.cc:302] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c21ea80db8bb4c9692a1ee2abdcdc485, e243ef1700264f1f986d1875c61e5fef; no voters:
I20260810 04:46:20.714393 29440 raft_consensus.cc:2781] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:20.714527 29440 raft_consensus.cc:686] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 LEADER]: Becoming Leader. State: Replica: c21ea80db8bb4c9692a1ee2abdcdc485, State: Running, Role: LEADER
I20260810 04:46:20.714600 29440 consensus_queue.cc:237] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.714967 29496 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: TxnStatusTablet state changed. Reason: New leader c21ea80db8bb4c9692a1ee2abdcdc485. Latest consensus state: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.715063 29496 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:46:20.715296 29501 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:46:20.715337 28521 catalog_manager.cc:5495] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 reported cstate change: term changed from 0 to 1, leader changed from <none> to c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } health_report { overall_health: UNKNOWN } } }
I20260810 04:46:20.715435 29501 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:46:20.751366 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.748850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.14.237.2:36575)
0810 04:46:20.748922 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.748923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.748934 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:46:20.748996 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.748997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.749025 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.749096 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:20.749273 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.749750 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.750510 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.750707 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.750787 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.751032 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:20.751134 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:20.751216 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.751267 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.751289 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:46:20.751385 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.748749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:36575 (local address 127.0.0.1:58804)
0810 04:46:20.748824 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.748843 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.748911 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.749116 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.749118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.749119 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:46:20.749178 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.749180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.749841 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.749842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.750424 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.750426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.750787 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.750789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.750899 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.750919 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:46:20.751203 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:46:20.751215 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.751264 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.751316 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:46:20.752679 29310 raft_consensus.cc:1261] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Refusing update from remote peer c21ea80db8bb4c9692a1ee2abdcdc485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:20.752847 28809 raft_consensus.cc:1261] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Refusing update from remote peer c21ea80db8bb4c9692a1ee2abdcdc485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:20.752981 29440 consensus_queue.cc:1035] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:20.753122 29496 consensus_queue.cc:1035] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:20.753777 29439 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: TxnStatusTablet state changed. Reason: New leader c21ea80db8bb4c9692a1ee2abdcdc485. Latest consensus state: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.753870 29439 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:46:20.753813 29436 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: TxnStatusTablet state changed. Reason: New leader c21ea80db8bb4c9692a1ee2abdcdc485. Latest consensus state: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.753896 29436 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:46:20.754873 29496 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.754982 29496 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:46:20.755043 29448 mvcc.cc:205] Tried to move back new op lower bound from 7316837092360695808 to 7316837092207472640. Current Snapshot: MvccSnapshot[applied={T|T < 7316837092360695808 or (T in {7316837092360695808})}]
I20260810 04:46:20.755282 29452 mvcc.cc:205] Tried to move back new op lower bound from 7316837092360695808 to 7316837092207472640. Current Snapshot: MvccSnapshot[applied={T|T < 7316837092360695808}]
I20260810 04:46:20.755395 29439 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.755471 29439 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P 03388ea92e124181a960c753236da9f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:46:20.755473 29496 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.755548 29496 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P c21ea80db8bb4c9692a1ee2abdcdc485: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:46:20.755862 29447 mvcc.cc:205] Tried to move back new op lower bound from 7316837092360695808 to 7316837092207472640. Current Snapshot: MvccSnapshot[applied={T|T < 7316837092360695808}]
I20260810 04:46:20.755959 29436 tablet_replica.cc:439] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } } }
I20260810 04:46:20.756053 29436 tablet_replica.cc:442] T f7a4e4ae90ac40fa88bcd1a19b505185 P e243ef1700264f1f986d1875c61e5fef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:46:20.760008 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.757621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.14.237.3:36347)
0810 04:46:20.757703 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.757706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.757714 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:46:20.757777 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.757778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.757795 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.757883 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:20.757994 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.758457 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.759200 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.759423 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.759503 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.759669 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:20.759778 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:20.759855 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.759915 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.759933 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 04:46:20.765161 29426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.762568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:37759 (local address 127.14.237.2:36575)
0810 04:46:20.762621 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.762623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.762723 (+ 100us) server_negotiation.cc:407] Connection header received
0810 04:46:20.762794 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.762794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.762802 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.762846 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:20.762992 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.763448 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.764540 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.764708 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.764754 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.764967 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:20.764990 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:20.765022 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.765101 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.765116 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:46:20.765272 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.762467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:36575 (local address 127.14.237.3:37759)
0810 04:46:20.762648 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.762660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.762723 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.762865 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.762866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.762867 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:46:20.762935 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.762936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.763523 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.763524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.764453 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.764456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.764794 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.764795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.764861 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.764887 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:46:20.765043 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:46:20.765045 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.765108 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.765154 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260810 04:46:20.767554 29050 raft_consensus.cc:1261] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Refusing update from remote peer 03388ea92e124181a960c753236da9f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:20.767812 28809 raft_consensus.cc:1261] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Refusing update from remote peer 03388ea92e124181a960c753236da9f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:20.767843 29439 consensus_queue.cc:1035] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:20.768011 29476 consensus_queue.cc:1035] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:20.776912 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.774032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:58817 (local address 127.14.237.3:36347)
0810 04:46:20.774075 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.774077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.774142 (+ 65us) server_negotiation.cc:407] Connection header received
0810 04:46:20.774234 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.774235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.774244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.774270 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:20.774430 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.775140 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.776210 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.776478 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.776519 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.776718 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:20.776747 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:20.776794 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.776847 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.776864 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:46:20.776990 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.773846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:36347 (local address 127.14.237.2:58817)
0810 04:46:20.774071 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:20.774081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:20.774168 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:20.774323 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:20.774324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:20.774325 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:46:20.774365 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.774366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.775240 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.775241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.776094 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:20.776108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.776574 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.776575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:20.776629 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.776642 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:46:20.776797 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:46:20.776799 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:46:20.776845 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 04:46:20.776894 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260810 04:46:20.785813 28522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:20.785926 28522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:20.786609 28522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: Sending DeleteTablet for 3 replicas of tablet 9b13c8caa915454d9bb6b5e2594c8ae1
I20260810 04:46:20.786872 28764 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b13c8caa915454d9bb6b5e2594c8ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:20 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:32958
I20260810 04:46:20.786952 29264 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b13c8caa915454d9bb6b5e2594c8ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:20 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:43480
I20260810 04:46:20.787038 29468 tablet_replica.cc:330] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: stopping tablet replica
I20260810 04:46:20.787115 29468 raft_consensus.cc:2229] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:20.787153 29468 raft_consensus.cc:2258] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:20.787314 29469 tablet_replica.cc:330] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: stopping tablet replica
I20260810 04:46:20.787381 29469 raft_consensus.cc:2229] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:20.787463 29469 raft_consensus.cc:2258] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:20.787770 29468 ts_tablet_manager.cc:1905] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:20.787874 29469 ts_tablet_manager.cc:1905] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:20.789527 29468 ts_tablet_manager.cc:1918] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:46:20.789584 29468 log.cc:1196] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/9b13c8caa915454d9bb6b5e2594c8ae1
I20260810 04:46:20.789702 29005 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b13c8caa915454d9bb6b5e2594c8ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:20 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:58788
I20260810 04:46:20.789790 29465 tablet_replica.cc:330] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: stopping tablet replica
I20260810 04:46:20.789798 29468 ts_tablet_manager.cc:1939] T 9b13c8caa915454d9bb6b5e2594c8ae1 P e243ef1700264f1f986d1875c61e5fef: Deleting consensus metadata
I20260810 04:46:20.789834 29465 raft_consensus.cc:2229] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:20.789868 29465 raft_consensus.cc:2258] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:20.790171 29465 ts_tablet_manager.cc:1905] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:20.790223 28492 catalog_manager.cc:4841] TS e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607): tablet 9b13c8caa915454d9bb6b5e2594c8ae1 (table test-table [id=6af81cad837e46c1a32c3c82a1bf1226]) successfully deleted
I20260810 04:46:20.791761 29465 ts_tablet_manager.cc:1918] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:46:20.791848 29465 log.cc:1196] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/9b13c8caa915454d9bb6b5e2594c8ae1
I20260810 04:46:20.792071 29465 ts_tablet_manager.cc:1939] T 9b13c8caa915454d9bb6b5e2594c8ae1 P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting consensus metadata
I20260810 04:46:20.792408 28497 catalog_manager.cc:4841] TS c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575): tablet 9b13c8caa915454d9bb6b5e2594c8ae1 (table test-table [id=6af81cad837e46c1a32c3c82a1bf1226]) successfully deleted
I20260810 04:46:20.795544 29469 ts_tablet_manager.cc:1918] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:46:20.795611 29469 log.cc:1196] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/9b13c8caa915454d9bb6b5e2594c8ae1
I20260810 04:46:20.795851 29469 ts_tablet_manager.cc:1939] T 9b13c8caa915454d9bb6b5e2594c8ae1 P 03388ea92e124181a960c753236da9f1: Deleting consensus metadata
I20260810 04:46:20.796224 28495 catalog_manager.cc:4841] TS 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347): tablet 9b13c8caa915454d9bb6b5e2594c8ae1 (table test-table [id=6af81cad837e46c1a32c3c82a1bf1226]) successfully deleted
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:46:20.822805 28624 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:20.815157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.14.237.62:42627)
0810 04:46:20.815206 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.815209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.815377 (+ 168us) server_negotiation.cc:407] Connection header received
0810 04:46:20.815458 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.815459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.815472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.815542 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:20.816285 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.817083 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.818254 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.818578 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.818651 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.822727 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35206: BlockingRecv error: recv got EOF from 127.0.0.1:35206 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 10 04:46:20 dist-test-slave-8hhm krb5kdc[28439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM
I20260810 04:46:20.832983 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:20.824001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.14.237.62:42627)
0810 04:46:20.824039 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:20.824041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:20.824121 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:46:20.824190 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:20.824191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:20.824200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:20.824262 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:20.824854 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.825682 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.826672 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:20.826980 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:20.827052 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:20.829810 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:20.829812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:20.829812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:20.829821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:20.831576 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.832037 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.832049 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.832050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.832153 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:20.832394 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:20.832395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:20.832396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:20.832542 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:20.832654 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:20.832790 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 04:46:20.832877 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:46:20.835098 28522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:46:20.835268 28522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:20.836673 28764 tablet_service.cc:1467] Processing CreateTablet for tablet 1879f93bd9ed4922925a6aa5b45a3c38 (DEFAULT_TABLE table=test-table [id=7ce83d5392b44e68bae363d81e53b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.836719 29005 tablet_service.cc:1467] Processing CreateTablet for tablet 1879f93bd9ed4922925a6aa5b45a3c38 (DEFAULT_TABLE table=test-table [id=7ce83d5392b44e68bae363d81e53b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.836831 29005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1879f93bd9ed4922925a6aa5b45a3c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.836833 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1879f93bd9ed4922925a6aa5b45a3c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.836800 29264 tablet_service.cc:1467] Processing CreateTablet for tablet 1879f93bd9ed4922925a6aa5b45a3c38 (DEFAULT_TABLE table=test-table [id=7ce83d5392b44e68bae363d81e53b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:20.836932 29264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1879f93bd9ed4922925a6aa5b45a3c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:20.838266 29429 tablet_bootstrap.cc:492] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: Bootstrap starting.
I20260810 04:46:20.838390 29428 tablet_bootstrap.cc:492] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: Bootstrap starting.
I20260810 04:46:20.838770 29428 tablet_bootstrap.cc:654] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.839015 29430 tablet_bootstrap.cc:492] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: Bootstrap starting.
I20260810 04:46:20.839053 29429 tablet_bootstrap.cc:654] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.839676 29430 tablet_bootstrap.cc:654] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:20.839846 29429 tablet_bootstrap.cc:492] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: No bootstrap required, opened a new log
I20260810 04:46:20.839890 29429 ts_tablet_manager.cc:1397] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:20.840032 29429 raft_consensus.cc:348] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.840098 29429 raft_consensus.cc:374] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.840119 29429 raft_consensus.cc:729] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03388ea92e124181a960c753236da9f1, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.840163 29429 consensus_queue.cc:260] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.840253 29429 ts_tablet_manager.cc:1428] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.840767 29430 tablet_bootstrap.cc:492] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: No bootstrap required, opened a new log
I20260810 04:46:20.840792 29428 tablet_bootstrap.cc:492] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: No bootstrap required, opened a new log
I20260810 04:46:20.840835 29430 ts_tablet_manager.cc:1397] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:46:20.840845 29428 ts_tablet_manager.cc:1397] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:20.840958 29428 raft_consensus.cc:348] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.840986 29430 raft_consensus.cc:348] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.841029 29428 raft_consensus.cc:374] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.841048 29430 raft_consensus.cc:374] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:20.841054 29428 raft_consensus.cc:729] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e243ef1700264f1f986d1875c61e5fef, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.841068 29430 raft_consensus.cc:729] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21ea80db8bb4c9692a1ee2abdcdc485, State: Initialized, Role: FOLLOWER
I20260810 04:46:20.841123 29430 consensus_queue.cc:260] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.841123 29428 consensus_queue.cc:260] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:20.841203 29430 ts_tablet_manager.cc:1428] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:20.842132 29428 ts_tablet_manager.cc:1428] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:46:21.206892 29531 raft_consensus.cc:482] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:21.207003 29531 raft_consensus.cc:504] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.207226 29531 leader_election.cc:288] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607), 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347)
I20260810 04:46:21.207535 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1879f93bd9ed4922925a6aa5b45a3c38" candidate_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03388ea92e124181a960c753236da9f1" is_pre_election: true
I20260810 04:46:21.207535 28809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1879f93bd9ed4922925a6aa5b45a3c38" candidate_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e243ef1700264f1f986d1875c61e5fef" is_pre_election: true
I20260810 04:46:21.207653 29310 raft_consensus.cc:2454] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c21ea80db8bb4c9692a1ee2abdcdc485 in term 0.
I20260810 04:46:21.207654 28809 raft_consensus.cc:2454] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c21ea80db8bb4c9692a1ee2abdcdc485 in term 0.
I20260810 04:46:21.207815 28929 leader_election.cc:302] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c21ea80db8bb4c9692a1ee2abdcdc485, e243ef1700264f1f986d1875c61e5fef; no voters:
I20260810 04:46:21.207930 29531 raft_consensus.cc:2781] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:21.207973 29531 raft_consensus.cc:482] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:21.207993 29531 raft_consensus.cc:3037] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:21.208712 29531 raft_consensus.cc:504] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.208858 29531 leader_election.cc:288] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [CANDIDATE]: Term 1 election: Requested vote from peers e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607), 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347)
I20260810 04:46:21.209012 28809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1879f93bd9ed4922925a6aa5b45a3c38" candidate_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e243ef1700264f1f986d1875c61e5fef"
I20260810 04:46:21.209101 28809 raft_consensus.cc:3037] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:21.209069 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1879f93bd9ed4922925a6aa5b45a3c38" candidate_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03388ea92e124181a960c753236da9f1"
I20260810 04:46:21.209188 29310 raft_consensus.cc:3037] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:21.209797 28809 raft_consensus.cc:2454] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c21ea80db8bb4c9692a1ee2abdcdc485 in term 1.
I20260810 04:46:21.209997 29310 raft_consensus.cc:2454] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c21ea80db8bb4c9692a1ee2abdcdc485 in term 1.
I20260810 04:46:21.209992 28929 leader_election.cc:302] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c21ea80db8bb4c9692a1ee2abdcdc485, e243ef1700264f1f986d1875c61e5fef; no voters:
I20260810 04:46:21.210105 29531 raft_consensus.cc:2781] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:21.210156 29531 raft_consensus.cc:686] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 LEADER]: Becoming Leader. State: Replica: c21ea80db8bb4c9692a1ee2abdcdc485, State: Running, Role: LEADER
I20260810 04:46:21.210211 29531 consensus_queue.cc:237] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.210898 28520 catalog_manager.cc:5495] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 reported cstate change: term changed from 0 to 1, leader changed from <none> to c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } health_report { overall_health: UNKNOWN } } }
I20260810 04:46:21.284891 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:21.282686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.14.237.2:36575)
0810 04:46:21.282802 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:21.282804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:21.282813 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:21.282846 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:21.282847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:21.282857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:21.282925 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:21.283123 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:21.283588 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:21.284290 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:21.284486 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:21.284543 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:21.284663 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:21.284688 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:21.284730 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:21.284757 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 04:46:21.284773 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260810 04:46:21.285876 28809 raft_consensus.cc:1261] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Refusing update from remote peer c21ea80db8bb4c9692a1ee2abdcdc485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:21.285861 29310 raft_consensus.cc:1261] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Refusing update from remote peer c21ea80db8bb4c9692a1ee2abdcdc485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:21.286127 29531 consensus_queue.cc:1035] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:21.286258 29496 consensus_queue.cc:1035] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:21.289192 28520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:21.289335 28520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:21.289959 28520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: Sending DeleteTablet for 3 replicas of tablet 1879f93bd9ed4922925a6aa5b45a3c38
I20260810 04:46:21.290267 29264 tablet_service.cc:1514] Processing DeleteTablet for tablet 1879f93bd9ed4922925a6aa5b45a3c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:21 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:43480
I20260810 04:46:21.290380 29469 tablet_replica.cc:330] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: stopping tablet replica
I20260810 04:46:21.290366 29005 tablet_service.cc:1514] Processing DeleteTablet for tablet 1879f93bd9ed4922925a6aa5b45a3c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:21 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:58788
I20260810 04:46:21.290426 29469 raft_consensus.cc:2229] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:21.290469 29469 raft_consensus.cc:2258] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:21.290457 28764 tablet_service.cc:1514] Processing DeleteTablet for tablet 1879f93bd9ed4922925a6aa5b45a3c38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:21 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:32958
I20260810 04:46:21.290585 29465 tablet_replica.cc:330] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: stopping tablet replica
I20260810 04:46:21.290650 29465 raft_consensus.cc:2229] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:21.290599 29585 tablet_replica.cc:330] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: stopping tablet replica
I20260810 04:46:21.290747 29465 raft_consensus.cc:2258] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:21.290782 29469 ts_tablet_manager.cc:1905] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:21.290766 29585 raft_consensus.cc:2229] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:21.290928 29585 raft_consensus.cc:2258] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:21.291105 28520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:46:21.291257 28520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:21.291462 29585 ts_tablet_manager.cc:1905] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:21.291816 29465 ts_tablet_manager.cc:1905] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:21.292622 29469 ts_tablet_manager.cc:1918] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:21.292698 29469 log.cc:1196] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/1879f93bd9ed4922925a6aa5b45a3c38
I20260810 04:46:21.292927 29469 ts_tablet_manager.cc:1939] T 1879f93bd9ed4922925a6aa5b45a3c38 P 03388ea92e124181a960c753236da9f1: Deleting consensus metadata
I20260810 04:46:21.293272 28495 catalog_manager.cc:4841] TS 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347): tablet 1879f93bd9ed4922925a6aa5b45a3c38 (table test-table [id=7ce83d5392b44e68bae363d81e53b98d]) successfully deleted
I20260810 04:46:21.293507 29465 ts_tablet_manager.cc:1918] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:21.293576 29465 log.cc:1196] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/1879f93bd9ed4922925a6aa5b45a3c38
I20260810 04:46:21.293710 29585 ts_tablet_manager.cc:1918] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:21.293798 29585 log.cc:1196] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/1879f93bd9ed4922925a6aa5b45a3c38
I20260810 04:46:21.293802 29465 ts_tablet_manager.cc:1939] T 1879f93bd9ed4922925a6aa5b45a3c38 P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting consensus metadata
I20260810 04:46:21.294016 29585 ts_tablet_manager.cc:1939] T 1879f93bd9ed4922925a6aa5b45a3c38 P e243ef1700264f1f986d1875c61e5fef: Deleting consensus metadata
I20260810 04:46:21.294121 28497 catalog_manager.cc:4841] TS c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575): tablet 1879f93bd9ed4922925a6aa5b45a3c38 (table test-table [id=7ce83d5392b44e68bae363d81e53b98d]) successfully deleted
I20260810 04:46:21.294438 28492 catalog_manager.cc:4841] TS e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607): tablet 1879f93bd9ed4922925a6aa5b45a3c38 (table test-table [id=7ce83d5392b44e68bae363d81e53b98d]) successfully deleted
I20260810 04:46:21.295045 28764 tablet_service.cc:1467] Processing CreateTablet for tablet 2b93a156143247668e8386405865267b (DEFAULT_TABLE table=test-table [id=35185f10109548aaae11ed00c562de1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:21.295100 29264 tablet_service.cc:1467] Processing CreateTablet for tablet 2b93a156143247668e8386405865267b (DEFAULT_TABLE table=test-table [id=35185f10109548aaae11ed00c562de1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:21.295143 29005 tablet_service.cc:1467] Processing CreateTablet for tablet 2b93a156143247668e8386405865267b (DEFAULT_TABLE table=test-table [id=35185f10109548aaae11ed00c562de1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:21.295161 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b93a156143247668e8386405865267b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:21.295214 29264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b93a156143247668e8386405865267b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:21.295241 29005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b93a156143247668e8386405865267b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:21.296309 29428 tablet_bootstrap.cc:492] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef: Bootstrap starting.
I20260810 04:46:21.296403 29430 tablet_bootstrap.cc:492] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: Bootstrap starting.
I20260810 04:46:21.296403 29429 tablet_bootstrap.cc:492] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: Bootstrap starting.
I20260810 04:46:21.296659 29428 tablet_bootstrap.cc:654] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:21.296715 29430 tablet_bootstrap.cc:654] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:21.296715 29429 tablet_bootstrap.cc:654] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:21.297361 29430 tablet_bootstrap.cc:492] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: No bootstrap required, opened a new log
I20260810 04:46:21.297418 29430 ts_tablet_manager.cc:1397] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:21.297550 29430 raft_consensus.cc:348] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.297640 29430 raft_consensus.cc:374] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:21.297663 29430 raft_consensus.cc:729] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21ea80db8bb4c9692a1ee2abdcdc485, State: Initialized, Role: FOLLOWER
I20260810 04:46:21.297709 29430 consensus_queue.cc:260] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.297797 29428 tablet_bootstrap.cc:492] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef: No bootstrap required, opened a new log
I20260810 04:46:21.297813 29430 ts_tablet_manager.cc:1428] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:21.297840 29428 ts_tablet_manager.cc:1397] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:21.297938 29428 raft_consensus.cc:348] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.297997 29428 raft_consensus.cc:374] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:21.298017 29428 raft_consensus.cc:729] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e243ef1700264f1f986d1875c61e5fef, State: Initialized, Role: FOLLOWER
I20260810 04:46:21.298038 29429 tablet_bootstrap.cc:492] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: No bootstrap required, opened a new log
I20260810 04:46:21.298060 29428 consensus_queue.cc:260] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.298079 29429 ts_tablet_manager.cc:1397] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:46:21.298143 29428 ts_tablet_manager.cc:1428] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:21.298213 29429 raft_consensus.cc:348] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.298285 29429 raft_consensus.cc:374] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:21.298311 29429 raft_consensus.cc:729] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03388ea92e124181a960c753236da9f1, State: Initialized, Role: FOLLOWER
I20260810 04:46:21.298357 29429 consensus_queue.cc:260] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.298444 29429 ts_tablet_manager.cc:1428] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:21.324249 29529 raft_consensus.cc:482] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:21.324369 29529 raft_consensus.cc:504] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.324563 29529 leader_election.cc:288] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607), c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575)
I20260810 04:46:21.324747 29050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b93a156143247668e8386405865267b" candidate_uuid: "03388ea92e124181a960c753236da9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" is_pre_election: true
I20260810 04:46:21.324745 28809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b93a156143247668e8386405865267b" candidate_uuid: "03388ea92e124181a960c753236da9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e243ef1700264f1f986d1875c61e5fef" is_pre_election: true
I20260810 04:46:21.324842 29050 raft_consensus.cc:2454] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03388ea92e124181a960c753236da9f1 in term 0.
I20260810 04:46:21.324843 28809 raft_consensus.cc:2454] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03388ea92e124181a960c753236da9f1 in term 0.
I20260810 04:46:21.325001 29185 leader_election.cc:302] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03388ea92e124181a960c753236da9f1, c21ea80db8bb4c9692a1ee2abdcdc485; no voters:
I20260810 04:46:21.325098 29529 raft_consensus.cc:2781] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:21.325135 29529 raft_consensus.cc:482] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:21.325150 29529 raft_consensus.cc:3037] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:21.325888 29529 raft_consensus.cc:504] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.326025 29529 leader_election.cc:288] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [CANDIDATE]: Term 1 election: Requested vote from peers e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607), c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575)
I20260810 04:46:21.326164 29050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b93a156143247668e8386405865267b" candidate_uuid: "03388ea92e124181a960c753236da9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485"
I20260810 04:46:21.326251 29050 raft_consensus.cc:3037] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:21.326210 28809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b93a156143247668e8386405865267b" candidate_uuid: "03388ea92e124181a960c753236da9f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e243ef1700264f1f986d1875c61e5fef"
I20260810 04:46:21.326313 28809 raft_consensus.cc:3037] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:21.326900 28809 raft_consensus.cc:2454] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03388ea92e124181a960c753236da9f1 in term 1.
I20260810 04:46:21.327006 29050 raft_consensus.cc:2454] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03388ea92e124181a960c753236da9f1 in term 1.
I20260810 04:46:21.327075 29179 leader_election.cc:302] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03388ea92e124181a960c753236da9f1, e243ef1700264f1f986d1875c61e5fef; no voters:
I20260810 04:46:21.327178 29529 raft_consensus.cc:2781] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:21.327291 29529 raft_consensus.cc:686] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 1 LEADER]: Becoming Leader. State: Replica: 03388ea92e124181a960c753236da9f1, State: Running, Role: LEADER
I20260810 04:46:21.327351 29529 consensus_queue.cc:237] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } }
I20260810 04:46:21.328011 28522 catalog_manager.cc:5495] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03388ea92e124181a960c753236da9f1 (127.14.237.3). New cstate: current_term: 1 leader_uuid: "03388ea92e124181a960c753236da9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03388ea92e124181a960c753236da9f1" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 36347 } health_report { overall_health: HEALTHY } } }
I20260810 04:46:21.338515 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:21.336222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.14.237.3:36347)
0810 04:46:21.336389 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:21.336391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:21.336398 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:46:21.336421 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:21.336425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:21.336435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:21.336510 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:21.336615 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:21.337082 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:21.337844 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:21.338051 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:21.338125 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:21.338266 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:21.338301 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:21.338353 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:21.338413 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 04:46:21.338433 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260810 04:46:21.340677 29050 raft_consensus.cc:1261] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Refusing update from remote peer 03388ea92e124181a960c753236da9f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:21.340679 28809 raft_consensus.cc:1261] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Refusing update from remote peer 03388ea92e124181a960c753236da9f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:21.340878 29595 consensus_queue.cc:1035] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e243ef1700264f1f986d1875c61e5fef" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:21.340957 29529 consensus_queue.cc:1035] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c21ea80db8bb4c9692a1ee2abdcdc485" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 36575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:21.354745 28522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:21.354871 28522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:21.355623 28522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c47f9724d3964122b46d3ffc2402cb65: Sending DeleteTablet for 3 replicas of tablet 2b93a156143247668e8386405865267b
I20260810 04:46:21.355892 28764 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b93a156143247668e8386405865267b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:21 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:32958
I20260810 04:46:21.355939 29005 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b93a156143247668e8386405865267b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:21 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:58788
I20260810 04:46:21.355989 29264 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b93a156143247668e8386405865267b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:21 UTC) from {username='oryx', principal='oryx/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:43480
I20260810 04:46:21.356027 29585 tablet_replica.cc:330] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef: stopping tablet replica
I20260810 04:46:21.356081 29585 raft_consensus.cc:2229] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:21.356096 29465 tablet_replica.cc:330] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: stopping tablet replica
I20260810 04:46:21.356119 29585 raft_consensus.cc:2258] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:21.356151 29465 raft_consensus.cc:2229] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:21.356187 29465 raft_consensus.cc:2258] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:21.356298 29469 tablet_replica.cc:330] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: stopping tablet replica
I20260810 04:46:21.356355 29469 raft_consensus.cc:2229] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:21.356412 29469 raft_consensus.cc:2258] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:21.356439 29585 ts_tablet_manager.cc:1905] T 2b93a156143247668e8386405865267b P e243ef1700264f1f986d1875c61e5fef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:21.356685 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 28661
I20260810 04:46:21.356824 29465 ts_tablet_manager.cc:1905] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:21.364882 29465 ts_tablet_manager.cc:1918] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:46:21.364949 29465 log.cc:1196] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/2b93a156143247668e8386405865267b
I20260810 04:46:21.365163 29465 ts_tablet_manager.cc:1939] T 2b93a156143247668e8386405865267b P c21ea80db8bb4c9692a1ee2abdcdc485: Deleting consensus metadata
I20260810 04:46:21.365558 28497 catalog_manager.cc:4841] TS c21ea80db8bb4c9692a1ee2abdcdc485 (127.14.237.2:36575): tablet 2b93a156143247668e8386405865267b (table test-table [id=35185f10109548aaae11ed00c562de1c]) successfully deleted
I20260810 04:46:21.366703 29469 ts_tablet_manager.cc:1905] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:21.368376 29469 ts_tablet_manager.cc:1918] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:46:21.368439 29469 log.cc:1196] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/2b93a156143247668e8386405865267b
I20260810 04:46:21.368647 29469 ts_tablet_manager.cc:1939] T 2b93a156143247668e8386405865267b P 03388ea92e124181a960c753236da9f1: Deleting consensus metadata
I20260810 04:46:21.369453 28495 catalog_manager.cc:4841] TS 03388ea92e124181a960c753236da9f1 (127.14.237.3:36347): tablet 2b93a156143247668e8386405865267b (table test-table [id=35185f10109548aaae11ed00c562de1c]) successfully deleted
W20260810 04:46:21.373811 28492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:42607 (error 108)
I20260810 04:46:21.374480 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:21.374264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42607 (local address 127.0.0.1:44548)
0810 04:46:21.374381 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:21.374429 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260810 04:46:21.374590 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 28901
W20260810 04:46:21.378180 28492 catalog_manager.cc:4584] TS e243ef1700264f1f986d1875c61e5fef (127.14.237.1:42607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b93a156143247668e8386405865267b: Network error: Client connection negotiation failed: client connection to 127.14.237.1:42607: connect: Connection refused (error 111)
I20260810 04:46:21.390249 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 29155
I20260810 04:46:21.406519 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 28464
2026-08-10T04:46:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3313 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:21 dist-test-slave-8hhm krb5kdc[29612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:21 dist-test-slave-8hhm krb5kdc[29612](info): set up 2 sockets
Aug 10 04:46:21 dist-test-slave-8hhm krb5kdc[29612](info): commencing operation
krb5kdc: starting...
W20260810 04:46:22.685755 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:22 dist-test-slave-8hhm krb5kdc[29612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:22.805886 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:42023
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:42023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:22.891618 29626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:22.891825 29626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:22.891888 29626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:22.893041 29626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:22.893079 29626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:22.893092 29626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:22.893141 29626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:22.893164 29626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:22.894636 29626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:42023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:42023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:22.894891 29626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:22.895128 29626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:22.900161 29636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:22.900197 29638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:22.900455 29640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:22.900345 29626 server_base.cc:1034] running on GCE node
I20260810 04:46:22.900758 29626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:22.901010 29626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:22.902174 29626 hybrid_clock.cc:648] HybridClock initialized: now 1786337182902139 us; error 51 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T04:46:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1502 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:22 dist-test-slave-8hhm krb5kdc[29649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:22 dist-test-slave-8hhm krb5kdc[29649](info): set up 2 sockets
Aug 10 04:46:22 dist-test-slave-8hhm krb5kdc[29649](info): commencing operation
krb5kdc: starting...
W20260810 04:46:24.178436 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:24.300132 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38689
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:38689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:24.382792 29669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:24.382995 29669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:24.383016 29669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:24.384073 29669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:24.384107 29669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:24.384121 29669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:24.384135 29669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:24.384146 29669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:24.385690 29669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:38689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:24.385959 29669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:24.386200 29669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:24.391213 29677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:24.391486 29669 server_base.cc:1034] running on GCE node
W20260810 04:46:24.391551 29679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:24.391621 29675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:24.391930 29669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:24.392346 29669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:24.393507 29669 hybrid_clock.cc:648] HybridClock initialized: now 1786337184393484 us; error 35 us; skew 500 ppm
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:24.396850 29669 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:24.397457 29669 webserver.cc:460] Webserver started at http://127.14.237.62:44321/ using document root <none> and password file <none>
I20260810 04:46:24.397748 29669 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:24.397814 29669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:24.397945 29669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:24.399462 29669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "335e38dcece64ac1bb45450a493929da"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "d9e32d3098de73b6278083d92325f513"
server_key_iv: "f419c0bd877e2c0701937ff6f68bb98c"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.399915 29669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "335e38dcece64ac1bb45450a493929da"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "d9e32d3098de73b6278083d92325f513"
server_key_iv: "f419c0bd877e2c0701937ff6f68bb98c"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.401872 29669 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260810 04:46:24.406543 29690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.406795 29669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 04:46:24.406903 29669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "335e38dcece64ac1bb45450a493929da"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "d9e32d3098de73b6278083d92325f513"
server_key_iv: "f419c0bd877e2c0701937ff6f68bb98c"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.406996 29669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:24.446473 29669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:24.449420 29669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:24.449602 29669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:24.454527 29669 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:38689
I20260810 04:46:24.454648 29781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:38689 every 8 connection(s)
I20260810 04:46:24.454969 29669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:24.455202 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 29669
I20260810 04:46:24.455329 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:24.455545 15284 external_mini_cluster.cc:1428] Setting key f3c9071ab2f4599c0daaa9f3090fdf39
I20260810 04:46:24.456223 29782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:24.459405 29782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Bootstrap starting.
I20260810 04:46:24.460038 29782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:24.460343 29782 log.cc:824] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:24.461055 29782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: No bootstrap required, opened a new log
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:24.463418 29782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } }
I20260810 04:46:24.463599 29782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:24.463701 29782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335e38dcece64ac1bb45450a493929da, State: Initialized, Role: FOLLOWER
I20260810 04:46:24.463858 29782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } }
I20260810 04:46:24.463955 29782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:24.463994 29782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:24.464063 29782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:24.464756 29782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } }
I20260810 04:46:24.464896 29782 leader_election.cc:302] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 335e38dcece64ac1bb45450a493929da; no voters:
I20260810 04:46:24.465095 29782 leader_election.cc:288] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:24.465153 29791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:24.465329 29791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 1 LEADER]: Becoming Leader. State: Replica: 335e38dcece64ac1bb45450a493929da, State: Running, Role: LEADER
I20260810 04:46:24.465435 29782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:24.465437 29791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } }
I20260810 04:46:24.465785 29791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 335e38dcece64ac1bb45450a493929da. Latest consensus state: current_term: 1 leader_uuid: "335e38dcece64ac1bb45450a493929da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } } }
I20260810 04:46:24.465873 29791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:24.465818 29795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "335e38dcece64ac1bb45450a493929da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } } }
I20260810 04:46:24.466069 29795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:24.467510 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:24.456130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.14.237.62:38689)
0810 04:46:24.456329 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:24.456339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:24.456373 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:46:24.456974 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:24.456984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:24.457049 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:24.457238 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:24.458079 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.459395 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.460253 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.460726 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.460850 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:24.463555 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:24.463559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:24.463559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:24.463569 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:24.465704 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:24.466149 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:24.466152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:24.466152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:24.466267 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:24.466515 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:24.466516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:24.466517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:24.466860 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:24.467029 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:24.467166 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 04:46:24.467296 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
W20260810 04:46:24.467777 29815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:46:24.467836 29815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:46:24.468122 29801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:24.468312 29801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:24.469739 29801 catalog_manager.cc:1347] Generated new cluster ID: 65098169d81d4458b1122a1496b4847f
I20260810 04:46:24.469794 29801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:24.475436 29801 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:24.475934 29801 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:24.479879 29801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Generated new TSK 0
I20260810 04:46:24.480037 29801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:46:24.511803 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38689
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:24.593900 29826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:24.594138 29826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:24.594177 29826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:24.595387 29826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:24.595454 29826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:24.595497 29826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:24.596969 29826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:24.597270 29826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:24.597500 29826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:24.602548 29839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:24.602587 29842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:24.602603 29826 server_base.cc:1034] running on GCE node
W20260810 04:46:24.602555 29838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:24.602998 29826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:24.603252 29826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:24.604415 29826 hybrid_clock.cc:648] HybridClock initialized: now 1786337184604406 us; error 39 us; skew 500 ppm
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:24.610090 29826 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:46:24.610551 29826 webserver.cc:460] Webserver started at http://127.14.237.1:35209/ using document root <none> and password file <none>
I20260810 04:46:24.610827 29826 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:24.610900 29826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:24.611054 29826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:24.612200 29826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "8fb634806b474a1983791d8e899ea816"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "647ffd85e6d07df0c18901549bc45c5b"
server_key_iv: "97bac1091b6445d56b59477d5a6cb5aa"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.612576 29826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "8fb634806b474a1983791d8e899ea816"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "647ffd85e6d07df0c18901549bc45c5b"
server_key_iv: "97bac1091b6445d56b59477d5a6cb5aa"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.614037 29826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260810 04:46:24.614818 29851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.615000 29826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:24.615106 29826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "8fb634806b474a1983791d8e899ea816"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "647ffd85e6d07df0c18901549bc45c5b"
server_key_iv: "97bac1091b6445d56b59477d5a6cb5aa"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.615185 29826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:24.635965 29826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:24.637940 29826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:24.638109 29826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:24.638350 29826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:24.638676 29826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:24.638738 29826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.638775 29826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:24.638800 29826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.644640 29826 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:42295
I20260810 04:46:24.644693 30063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:42295 every 8 connection(s)
I20260810 04:46:24.645007 29826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:24.647583 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 29826
I20260810 04:46:24.647684 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:24.647909 15284 external_mini_cluster.cc:1428] Setting key 4e55d7afccfa57daeba32b7eb1ee7671
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:24.655478 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:24.645775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:38463 (local address 127.14.237.62:38689)
0810 04:46:24.645852 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:24.645854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:24.646034 (+ 180us) server_negotiation.cc:407] Connection header received
0810 04:46:24.646524 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:24.646526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:24.646535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:24.646594 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:24.647543 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.648294 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.649483 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.649713 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.649789 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:24.652167 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:24.652169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:24.652169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:24.652178 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:24.654023 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:24.654456 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:24.654458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:24.654458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:24.654529 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:24.654774 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:24.654775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:24.654776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:24.654935 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:24.655016 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:24.655261 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 04:46:24.655370 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:46:24.655503 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:24.645740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:38463)
0810 04:46:24.645901 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:24.645944 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:24.646445 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:24.646650 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:24.646653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:24.647233 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:24.647439 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:24.647444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.648395 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.648396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:24.649391 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:24.649394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.649812 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.649813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:24.649952 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:24.650508 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:24.650517 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:24.652069 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:24.654130 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:24.654137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:24.654138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:24.654375 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:24.654605 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:24.654605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:24.654605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:24.654705 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:24.655008 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:24.655013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:24.655125 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 04:46:24.655204 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:46:24.655340 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260810 04:46:24.656220 30066 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38689
I20260810 04:46:24.656344 30066 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:24.656565 30066 heartbeater.cc:511] Master 127.14.237.62:38689 requested a full tablet report, sending...
I20260810 04:46:24.657138 29720 ts_manager.cc:194] Registered new tserver with Master: 8fb634806b474a1983791d8e899ea816 (127.14.237.1:42295)
I20260810 04:46:24.657788 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:38463
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:46:24.676087 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38689
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:24.759059 30075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:24.759300 30075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:24.759338 30075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:24.760488 30075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:24.760568 30075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:24.760614 30075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:24.762066 30075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:24.762346 30075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:24.762578 30075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:24.767797 30091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:24.767812 30084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:24.767961 30075 server_base.cc:1034] running on GCE node
W20260810 04:46:24.767812 30087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:24.768263 30075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:24.768452 30075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:24.769582 30075 hybrid_clock.cc:648] HybridClock initialized: now 1786337184769565 us; error 41 us; skew 500 ppm
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:24.772049 30075 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:46:24.772508 30075 webserver.cc:460] Webserver started at http://127.14.237.2:43723/ using document root <none> and password file <none>
I20260810 04:46:24.772693 30075 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:24.772735 30075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:24.772825 30075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:24.773897 30075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "4aa4f4f1cfb64a1ea50aeba4aafddc8f"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "b6c426c88427f28e252b3cd81e9a3f2e"
server_key_iv: "4319c2cbe54fa12eacc2aa039f3fae4f"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.774235 30075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "4aa4f4f1cfb64a1ea50aeba4aafddc8f"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "b6c426c88427f28e252b3cd81e9a3f2e"
server_key_iv: "4319c2cbe54fa12eacc2aa039f3fae4f"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.775494 30075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:46:24.776146 30102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.776325 30075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:24.776414 30075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "4aa4f4f1cfb64a1ea50aeba4aafddc8f"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "b6c426c88427f28e252b3cd81e9a3f2e"
server_key_iv: "4319c2cbe54fa12eacc2aa039f3fae4f"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.776499 30075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:24.798316 30075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:24.800321 30075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:24.800467 30075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:24.800731 30075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:24.801056 30075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:24.801111 30075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.801162 30075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:24.801190 30075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.807065 30075 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:35883
I20260810 04:46:24.807111 30292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:35883 every 8 connection(s)
I20260810 04:46:24.807456 30075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:46:24.811713 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 30075
I20260810 04:46:24.811869 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:24.812100 15284 external_mini_cluster.cc:1428] Setting key 9cee0ce2ae0dd8a40f0116f234b01504
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:24.818014 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:24.808245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:55749 (local address 127.14.237.62:38689)
0810 04:46:24.808316 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:24.808318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:24.808511 (+ 193us) server_negotiation.cc:407] Connection header received
0810 04:46:24.808973 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:24.808974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:24.808984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:24.809031 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:24.809998 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.810473 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.811633 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.811863 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.811937 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:24.814806 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:24.814807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:24.814808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:24.814816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:24.816421 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:24.816885 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:24.816887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:24.816887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:24.816954 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:24.817188 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:24.817189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:24.817190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:24.817377 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:24.817456 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:24.817756 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 04:46:24.817889 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:46:24.818084 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:24.808205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:55749)
0810 04:46:24.808394 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:24.808431 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:24.808894 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:24.809101 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:24.809104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:24.809742 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:24.809923 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:24.809928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.810573 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.810574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:24.811550 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:24.811553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.811977 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.811979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:24.812172 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:24.812827 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:24.812835 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:24.814685 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:24.816582 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:24.816588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:24.816590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:24.816806 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:24.817023 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:24.817024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:24.817024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:24.817121 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:24.817451 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:24.817456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:24.817569 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 04:46:24.817716 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 04:46:24.817902 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260810 04:46:24.818603 30293 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38689
I20260810 04:46:24.818717 30293 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:24.818943 30293 heartbeater.cc:511] Master 127.14.237.62:38689 requested a full tablet report, sending...
I20260810 04:46:24.819371 29720 ts_manager.cc:194] Registered new tserver with Master: 4aa4f4f1cfb64a1ea50aeba4aafddc8f (127.14.237.2:35883)
I20260810 04:46:24.819748 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:55749
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:46:24.839982 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38689
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:24.926280 30306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:24.926476 30306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:24.926497 30306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:24.927699 30306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:24.927748 30306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:24.927812 30306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:24.929136 30306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:24.929400 30306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:24.929620 30306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:24.934864 30321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:24.934886 30317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:24.934911 30319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:24.934926 30306 server_base.cc:1034] running on GCE node
I20260810 04:46:24.935271 30306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:24.935541 30306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:24.936713 30306 hybrid_clock.cc:648] HybridClock initialized: now 1786337184936696 us; error 28 us; skew 500 ppm
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:24.939150 30306 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:46:24.939607 30306 webserver.cc:460] Webserver started at http://127.14.237.3:40171/ using document root <none> and password file <none>
I20260810 04:46:24.939834 30306 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:24.939903 30306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:24.940058 30306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:24.941129 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "9195dbde30dc4864bcfcfab2cde1c650"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "5d72b1037ff92cde89e36fc2f819ec80"
server_key_iv: "82a135c40e2786225e39ecf61488efe0"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.941561 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "9195dbde30dc4864bcfcfab2cde1c650"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "5d72b1037ff92cde89e36fc2f819ec80"
server_key_iv: "82a135c40e2786225e39ecf61488efe0"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.942915 30306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:24.943619 30331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.943778 30306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:24.943889 30306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "9195dbde30dc4864bcfcfab2cde1c650"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "5d72b1037ff92cde89e36fc2f819ec80"
server_key_iv: "82a135c40e2786225e39ecf61488efe0"
server_key_version: "encryptionkey@0"
I20260810 04:46:24.943960 30306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:24.952777 30306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:24.955327 30306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:24.955502 30306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:24.955737 30306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:24.956059 30306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:24.956116 30306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.956151 30306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:24.956177 30306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:24.962117 30306 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:33845
I20260810 04:46:24.962183 30520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:33845 every 8 connection(s)
I20260810 04:46:24.962518 30306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:24.965448 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 30306
I20260810 04:46:24.965544 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:24.965785 15284 external_mini_cluster.cc:1428] Setting key 77589b2955d306f4a3c945e8d233c6aa
Aug 10 04:46:24 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337184, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:24.972167 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:24.963264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:52475 (local address 127.14.237.62:38689)
0810 04:46:24.963322 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:24.963323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:24.963489 (+ 166us) server_negotiation.cc:407] Connection header received
0810 04:46:24.963940 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:24.963941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:24.963950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:24.964008 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:24.965048 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.965917 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.967407 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.967616 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.967714 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:24.969819 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:24.969821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:24.969821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:24.969827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:24.970976 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:24.971346 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:24.971347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:24.971348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:24.971402 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:24.971613 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:24.971614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:24.971614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:24.971709 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:24.971758 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:24.972025 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 04:46:24.972084 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:46:24.972306 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:24.963214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:52475)
0810 04:46:24.963369 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:24.963407 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:24.963858 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:24.964141 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:24.964164 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:24.964775 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:24.964962 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:24.964967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.966028 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.966033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:24.967304 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:24.967307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:24.967725 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:24.967726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:24.967856 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:24.968374 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:24.968382 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:24.969740 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:24.971048 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:24.971054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:24.971055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:24.971280 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:24.971462 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:24.971462 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:24.971462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:24.971557 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:24.971763 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:24.971767 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:24.971905 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 04:46:24.971998 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 04:46:24.972110 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260810 04:46:24.972800 30521 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38689
I20260810 04:46:24.972936 30521 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:24.973127 30521 heartbeater.cc:511] Master 127.14.237.62:38689 requested a full tablet report, sending...
I20260810 04:46:24.973546 29720 ts_manager.cc:194] Registered new tserver with Master: 9195dbde30dc4864bcfcfab2cde1c650 (127.14.237.3:33845)
I20260810 04:46:24.973908 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:52475
I20260810 04:46:24.977566 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab.
Entry for principal oryx/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab.
WARNING: no policy specified for oryx/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab.
Entry for principal oryx/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab.
WARNING: no policy specified for oryx/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab.
Entry for principal oryx/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab.
I20260810 04:46:25.049963 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 29826
I20260810 04:46:25.071763 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 30075
I20260810 04:46:25.097149 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 30306
I20260810 04:46:25.108233 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 29669
I20260810 04:46:25.119410 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38689
--webserver_interface=127.14.237.62
--webserver_port=44321
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:38689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:25.201145 30535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:25.201408 30535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:25.201447 30535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:25.202587 30535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:25.202664 30535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:25.202687 30535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:25.202708 30535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:25.202744 30535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:25.204236 30535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:38689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=44321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:25.204515 30535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:25.204753 30535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:25.209556 30548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:25.209581 30545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:25.209556 30550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:25.209815 30535 server_base.cc:1034] running on GCE node
I20260810 04:46:25.210126 30535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:25.210415 30535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:25.211584 30535 hybrid_clock.cc:648] HybridClock initialized: now 1786337185211569 us; error 43 us; skew 500 ppm
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337185, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:25.214404 30535 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:25.214788 30535 webserver.cc:460] Webserver started at http://127.14.237.62:44321/ using document root <none> and password file <none>
I20260810 04:46:25.214993 30535 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:25.215046 30535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:25.216285 30535 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:25.216934 30562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.217133 30535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:25.217206 30535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "335e38dcece64ac1bb45450a493929da"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "d9e32d3098de73b6278083d92325f513"
server_key_iv: "f419c0bd877e2c0701937ff6f68bb98c"
server_key_version: "encryptionkey@0"
I20260810 04:46:25.217595 30535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:25.252281 30535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:25.254251 30535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:25.254366 30535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:25.258459 30535 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:38689
I20260810 04:46:25.258517 30643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:38689 every 8 connection(s)
I20260810 04:46:25.258869 30535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:25.260419 30648 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:46:25.261114 30648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Bootstrap starting.
I20260810 04:46:25.264865 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 30535
I20260810 04:46:25.265200 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:42295
--local_ip_for_outbound_sockets=127.14.237.1
--tserver_master_addrs=127.14.237.62:38689
--webserver_port=35209
--webserver_interface=127.14.237.1
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.237.1
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:46:25.269682 30648 log.cc:824] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:25.271370 30648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:46:25.271659 30648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Bootstrap complete.
I20260810 04:46:25.274060 30648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } }
I20260810 04:46:25.274334 30648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335e38dcece64ac1bb45450a493929da, State: Initialized, Role: FOLLOWER
I20260810 04:46:25.274519 30648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } }
I20260810 04:46:25.274600 30648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:25.274653 30648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:25.274704 30648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:46:25.275708 30648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } }
I20260810 04:46:25.275837 30648 leader_election.cc:302] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 335e38dcece64ac1bb45450a493929da; no voters:
I20260810 04:46:25.276053 30648 leader_election.cc:288] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:46:25.276087 30656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:46:25.276206 30656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [term 2 LEADER]: Becoming Leader. State: Replica: 335e38dcece64ac1bb45450a493929da, State: Running, Role: LEADER
I20260810 04:46:25.276317 30648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:25.276299 30656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } }
I20260810 04:46:25.276561 30656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 335e38dcece64ac1bb45450a493929da. Latest consensus state: current_term: 2 leader_uuid: "335e38dcece64ac1bb45450a493929da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } } }
I20260810 04:46:25.276643 30656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:25.276566 30657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "335e38dcece64ac1bb45450a493929da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335e38dcece64ac1bb45450a493929da" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38689 } } }
I20260810 04:46:25.276798 30657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:25.276866 30662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:25.277416 30662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:25.277565 30662 catalog_manager.cc:1259] Loaded cluster ID: 65098169d81d4458b1122a1496b4847f
I20260810 04:46:25.277822 30662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:25.278585 30662 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:25.278971 30662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 335e38dcece64ac1bb45450a493929da: Loaded TSK: 0
I20260810 04:46:25.279256 30662 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:46:25.361959 30651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:25.362174 30651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:25.362197 30651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:25.363356 30651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:25.363399 30651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:25.363425 30651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:25.364866 30651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.1.keytab
--principal=oryx/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:42295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=35209
--tserver_master_addrs=127.14.237.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:25.365131 30651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:25.365393 30651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:25.370443 30687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:25.370457 30686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:25.370457 30693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:25.371074 30651 server_base.cc:1034] running on GCE node
I20260810 04:46:25.371284 30651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:25.371549 30651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:25.372747 30651 hybrid_clock.cc:648] HybridClock initialized: now 1786337185372732 us; error 27 us; skew 500 ppm
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337185, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:25.375846 30651 init.cc:377] Logged in from keytab as oryx/127.14.237.1@KRBTEST.COM (short username oryx)
I20260810 04:46:25.376299 30651 webserver.cc:460] Webserver started at http://127.14.237.1:35209/ using document root <none> and password file <none>
I20260810 04:46:25.376525 30651 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:25.376607 30651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:25.379829 30651 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 04:46:25.380568 30705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.380785 30651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:25.380890 30651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "8fb634806b474a1983791d8e899ea816"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "647ffd85e6d07df0c18901549bc45c5b"
server_key_iv: "97bac1091b6445d56b59477d5a6cb5aa"
server_key_version: "encryptionkey@0"
I20260810 04:46:25.381347 30651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:25.395756 30651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:25.398286 30651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:25.398468 30651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:25.398723 30651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:25.399096 30651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:25.399151 30651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.399194 30651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:25.399220 30651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.405592 30651 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:42295
I20260810 04:46:25.405658 30883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:42295 every 8 connection(s)
I20260810 04:46:25.406006 30651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:25.411320 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 30651
I20260810 04:46:25.411682 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:35883
--local_ip_for_outbound_sockets=127.14.237.2
--tserver_master_addrs=127.14.237.62:38689
--webserver_port=43723
--webserver_interface=127.14.237.2
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.237.2
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:25.416790 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.406892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:48205)
0810 04:46:25.407099 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.407151 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.407787 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.408219 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.408223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.409022 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.409323 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.409329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.410485 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.410486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.411755 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.411758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.412732 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.412734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.412891 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.413630 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.413642 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.416570 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
W20260810 04:46:25.417024 30884 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:38689 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.237.62:38689: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
W20260810 04:46:25.417363 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.407016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:48205 (local address 127.14.237.62:38689)
0810 04:46:25.407213 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.407220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.407249 (+ 29us) server_negotiation.cc:407] Connection header received
0810 04:46:25.407986 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.407991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.408033 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.408141 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.409453 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.410366 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.411879 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.412614 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.412745 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.416760 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:48205: BlockingRecv error: recv got EOF from 127.14.237.1:48205 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:25.423502 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.417247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:60555)
0810 04:46:25.417538 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.417549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.417593 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.417882 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.417884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.418068 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.418181 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.418184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.419121 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.419122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.419993 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.419996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.420515 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.420516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.420596 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.421123 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.421129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.423425 (+ 2296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":73,"threads_started":1}
W20260810 04:46:25.423854 30895 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.417417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:60555 (local address 127.14.237.62:38689)
0810 04:46:25.417737 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.417738 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.417745 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:46:25.417769 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.417770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.417781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.417845 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.418278 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.419022 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.420097 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.420427 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.420508 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.423755 (+ 3247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:60555: BlockingRecv error: recv got EOF from 127.14.237.1:60555 (error 108)
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":62,"threads_started":1}
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:25.429273 30894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.423776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:52387)
0810 04:46:25.423914 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.423924 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.423971 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.424219 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.424220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.424463 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.424515 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.424517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.425545 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.425545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.426106 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.426107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.426680 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.426682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.426758 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.427260 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.427272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.429170 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108}
W20260810 04:46:25.429425 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.423864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:52387 (local address 127.14.237.62:38689)
0810 04:46:25.424044 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.424047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.424058 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:46:25.424083 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.424084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.424100 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.424199 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.424602 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.425471 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.426185 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.426500 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.426618 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.429358 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:52387: BlockingRecv error: recv got EOF from 127.14.237.1:52387 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
W20260810 04:46:25.429454 30884 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:46:25.498737 30893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:25.498983 30893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:25.499022 30893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:25.500108 30893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:25.500181 30893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:25.500226 30893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:25.501628 30893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.2.keytab
--principal=oryx/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:35883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=43723
--tserver_master_addrs=127.14.237.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:25.501900 30893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:25.502139 30893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:25.507293 30907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:25.507274 30904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:25.507261 30912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:25.507309 30893 server_base.cc:1034] running on GCE node
I20260810 04:46:25.507702 30893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:25.507941 30893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:25.509085 30893 hybrid_clock.cc:648] HybridClock initialized: now 1786337185509083 us; error 65 us; skew 500 ppm
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337185, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:25.511552 30893 init.cc:377] Logged in from keytab as oryx/127.14.237.2@KRBTEST.COM (short username oryx)
I20260810 04:46:25.511956 30893 webserver.cc:460] Webserver started at http://127.14.237.2:43723/ using document root <none> and password file <none>
I20260810 04:46:25.512226 30893 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:25.512323 30893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:25.513790 30893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:46:25.514425 30934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.514606 30893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:25.514714 30893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "4aa4f4f1cfb64a1ea50aeba4aafddc8f"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "b6c426c88427f28e252b3cd81e9a3f2e"
server_key_iv: "4319c2cbe54fa12eacc2aa039f3fae4f"
server_key_version: "encryptionkey@0"
I20260810 04:46:25.515074 30893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:25.526618 30893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:25.528436 30893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:25.528590 30893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:25.528842 30893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:25.529139 30893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:25.529191 30893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.529278 30893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:25.529317 30893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.535147 30893 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:35883
I20260810 04:46:25.535197 31114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:35883 every 8 connection(s)
I20260810 04:46:25.535547 30893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:46:25.538477 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 30893
I20260810 04:46:25.538828 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:33845
--local_ip_for_outbound_sockets=127.14.237.3
--tserver_master_addrs=127.14.237.62:38689
--webserver_port=40171
--webserver_interface=127.14.237.3
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.237.3
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:46:25.543551 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.536219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:59293 (local address 127.14.237.62:38689)
0810 04:46:25.536310 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.536313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.536514 (+ 201us) server_negotiation.cc:407] Connection header received
0810 04:46:25.536956 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.536958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.536972 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.537022 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.537992 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.538477 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.539974 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.540156 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.540209 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.543473 (+ 3264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:59293: BlockingRecv error: recv got EOF from 127.14.237.2:59293 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:46:25.543551 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.536206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:59293)
0810 04:46:25.536384 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.536420 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.536875 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.537075 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.537078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.537732 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.537913 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.537918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.538593 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.538594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.539885 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.539888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.540239 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.540241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.540407 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.541008 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.541016 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.543342 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
W20260810 04:46:25.543852 31115 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:38689 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.237.62:38689: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:25.548779 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.544106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:55723)
0810 04:46:25.544235 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.544245 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.544284 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.544428 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.544428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.544544 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.544591 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.544592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.545551 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.545552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.546050 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.546051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.546552 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.546553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.546610 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.547060 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.547064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.548734 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260810 04:46:25.548946 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.544196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:55723 (local address 127.14.237.62:38689)
0810 04:46:25.544265 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.544267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.544279 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:46:25.544331 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.544332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.544349 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.544409 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.544663 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.545471 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.546133 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.546453 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.546533 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.548852 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:55723: BlockingRecv error: recv got EOF from 127.14.237.2:55723 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:25.553478 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.549033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:49281)
0810 04:46:25.549167 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.549177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.549234 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.549376 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.549376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.549477 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.549502 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.549503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.550397 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.550398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.550861 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.550862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.551303 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.551304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.551339 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.551756 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.551760 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.553433 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260810 04:46:25.553583 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.549175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:49281 (local address 127.14.237.62:38689)
0810 04:46:25.549235 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.549237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.549246 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:25.549293 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.549294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.549304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.549375 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.549572 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.550298 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.550942 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.551230 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.551297 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.553527 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:49281: BlockingRecv error: recv got EOF from 127.14.237.2:49281 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 04:46:25.553793 31115 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:46:25.625123 31118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:25.625449 31118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:25.625475 31118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:25.626681 31118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:25.626729 31118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:25.626780 31118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:25.628175 31118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/krb5kdc/oryx_127.14.237.3.keytab
--principal=oryx/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:33845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=40171
--tserver_master_addrs=127.14.237.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:25.628415 31118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:25.628646 31118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:25.633639 31128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:25.633651 31129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:25.633618 31131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:25.634030 31118 server_base.cc:1034] running on GCE node
I20260810 04:46:25.634279 31118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:25.634549 31118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:25.635713 31118 hybrid_clock.cc:648] HybridClock initialized: now 1786337185635708 us; error 39 us; skew 500 ppm
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337185, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:25.638223 31118 init.cc:377] Logged in from keytab as oryx/127.14.237.3@KRBTEST.COM (short username oryx)
I20260810 04:46:25.638643 31118 webserver.cc:460] Webserver started at http://127.14.237.3:40171/ using document root <none> and password file <none>
I20260810 04:46:25.638899 31118 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:25.638983 31118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:25.640311 31118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:25.640920 31142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.641088 31118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:25.641192 31118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "9195dbde30dc4864bcfcfab2cde1c650"
format_stamp: "Formatted at 2026-08-10 04:46:24 on dist-test-slave-8hhm"
server_key: "5d72b1037ff92cde89e36fc2f819ec80"
server_key_iv: "82a135c40e2786225e39ecf61488efe0"
server_key_version: "encryptionkey@0"
I20260810 04:46:25.641551 31118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:25.650722 31118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:25.652462 31118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:25.652635 31118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:25.652840 31118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:25.653139 31118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:25.653204 31118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.653264 31118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:25.653293 31118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:25.659276 31118 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:33845
I20260810 04:46:25.659312 31324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:33845 every 8 connection(s)
I20260810 04:46:25.659657 31118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:25.665364 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 31118
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:46:25.666870 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.660378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:34803 (local address 127.14.237.62:38689)
0810 04:46:25.660450 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.660452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.660669 (+ 217us) server_negotiation.cc:407] Connection header received
0810 04:46:25.661148 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.661149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.661158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.661272 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.662219 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.662695 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.663661 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.663852 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.663903 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.666816 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:34803: BlockingRecv error: recv got EOF from 127.14.237.3:34803 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:46:25.666937 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.660351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:34803)
0810 04:46:25.660527 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.660569 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.661058 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.661360 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.661364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.661937 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.662134 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.662138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.662775 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.662776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.663585 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.663588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.663927 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.663928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.664026 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.664499 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.664507 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.666739 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
W20260810 04:46:25.667124 31325 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:38689 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.237.62:38689: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:25.671626 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.667329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:33017)
0810 04:46:25.667381 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.667388 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.667446 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.667590 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.667591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.667705 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.667743 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.667744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.668319 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.668320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.668978 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.668980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.669399 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.669400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.669466 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.669882 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.669887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.671552 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:46:25.671674 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.667379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:33017 (local address 127.14.237.62:38689)
0810 04:46:25.667443 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.667445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.667454 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:25.667506 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.667507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.667515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.667584 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.667813 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.668239 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.669079 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.669327 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.669400 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.671614 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:33017: BlockingRecv error: recv got EOF from 127.14.237.3:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:46:25.672899 30895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.665862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.14.237.62:38689)
0810 04:46:25.665930 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.665932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.665992 (+ 60us) server_negotiation.cc:407] Connection header received
0810 04:46:25.666071 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.666073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.666084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.666134 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.666718 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.667308 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.668097 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.668308 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.668391 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.669712 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:25.669716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:25.669716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:25.669726 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:25.671388 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:25.671749 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:25.671752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:25.671753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:25.671880 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:25.672089 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:25.672090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:25.672090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:25.672419 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:25.672571 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:25.672676 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 04:46:25.672776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":40}
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:25 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:25.676687 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:25.672130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:56279)
0810 04:46:25.672236 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:25.672247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:25.672296 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:25.672403 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:25.672404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:25.672548 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:25.672579 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.672581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.673472 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.673473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.674116 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:25.674118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.674517 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.674518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:25.674561 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.674906 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:25.674911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:25.676648 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260810 04:46:25.676895 30889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:25.672188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:56279 (local address 127.14.237.62:38689)
0810 04:46:25.672246 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:25.672248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:25.672283 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:46:25.672348 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:25.672349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:25.672356 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:25.672399 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:25.672665 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.673387 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.674221 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:25.674421 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:25.674490 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:25.676710 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:56279: BlockingRecv error: recv got EOF from 127.14.237.3:56279 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 04:46:25.676961 31325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 04:46:26 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:26 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:26.434820 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:26.429877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:54759)
0810 04:46:26.430006 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:26.430017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:26.430067 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:26.430270 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:26.430271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:26.430450 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:26.430500 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:26.430502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.431145 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.431145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:26.431957 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:26.431960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.432320 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.432321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:26.432362 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:26.432874 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:26.432879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:26.434761 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
W20260810 04:46:26.434962 31333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:26.429951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:54759 (local address 127.14.237.62:38689)
0810 04:46:26.430126 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:26.430129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:26.430140 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:46:26.430172 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:26.430173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:26.430185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:26.430265 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:26.430576 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.431068 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.432041 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.432250 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.432312 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:26.434867 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:54759: BlockingRecv error: recv got EOF from 127.14.237.1:54759 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":142,"threads_started":1}
Aug 10 04:46:26 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:26 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:26.558923 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:26.554178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:49129)
0810 04:46:26.554341 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:26.554352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:26.554409 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:26.554541 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:26.554542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:26.554711 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:26.554769 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:26.554770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.555398 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.555398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:26.555930 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:26.555931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.556276 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.556277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:26.556315 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:26.556817 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:26.556822 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:26.558852 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260810 04:46:26.559033 31333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:26.554236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:49129 (local address 127.14.237.62:38689)
0810 04:46:26.554309 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:26.554311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:26.554410 (+ 99us) server_negotiation.cc:407] Connection header received
0810 04:46:26.554455 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:26.554455 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:26.554467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:26.554521 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:26.554833 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.555330 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.556018 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.556211 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.556263 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:26.558956 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:49129: BlockingRecv error: recv got EOF from 127.14.237.2:49129 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 04:46:26 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:26 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:26.682173 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:26.677383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:43981)
0810 04:46:26.677530 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:26.677545 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:26.677599 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:26.677731 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:26.677731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:26.677950 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:26.677996 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:26.677998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.678708 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.678709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:26.679240 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:26.679242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.679604 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.679605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:26.679665 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:26.680254 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:26.680260 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:26.682104 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
W20260810 04:46:26.682250 31333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:26.677445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:43981 (local address 127.14.237.62:38689)
0810 04:46:26.677517 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:26.677519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:26.677600 (+ 81us) server_negotiation.cc:407] Connection header received
0810 04:46:26.677643 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:26.677644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:26.677658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:26.677721 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:26.678101 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.678636 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.679313 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:26.679503 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:26.679579 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:26.682192 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:43981: BlockingRecv error: recv got EOF from 127.14.237.3:43981 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 04:46:27 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:27 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:27.440132 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:27.435381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:37961)
0810 04:46:27.435533 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:27.435546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:27.435603 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:27.435776 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:27.435777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:27.435951 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:27.436005 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:27.436006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.436709 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.436710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:27.437249 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:27.437251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.437630 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.437631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:27.437670 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:27.438131 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:27.438135 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:27.440074 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
W20260810 04:46:27.440311 31341 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:27.435437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:37961 (local address 127.14.237.62:38689)
0810 04:46:27.435623 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:27.435625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:27.435653 (+ 28us) server_negotiation.cc:407] Connection header received
0810 04:46:27.435678 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:27.435679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:27.435692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:27.435761 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:27.436131 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.436639 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.437359 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.437552 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.437630 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:27.440193 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:37961: BlockingRecv error: recv got EOF from 127.14.237.1:37961 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
Aug 10 04:46:27 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:27 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:27.564380 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:27.559544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:57697)
0810 04:46:27.559685 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:27.559695 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:27.559792 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:27.559959 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:27.559960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:27.560213 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:27.560273 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:27.560275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.560892 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.560893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:27.561528 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:27.561530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.561864 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.561865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:27.561903 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:27.562390 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:27.562394 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:27.564278 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
W20260810 04:46:27.564440 31341 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:27.559597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:57697 (local address 127.14.237.62:38689)
0810 04:46:27.559647 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:27.559649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:27.559836 (+ 187us) server_negotiation.cc:407] Connection header received
0810 04:46:27.559867 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:27.559868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:27.559887 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:27.559929 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:27.560350 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.560827 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.561610 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.561796 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.561852 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:27.564375 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:57697: BlockingRecv error: recv got EOF from 127.14.237.2:57697 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 10 04:46:27 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:27 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:27.687485 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:27.682751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:47877)
0810 04:46:27.682953 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:27.682968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:27.683070 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:27.683179 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:27.683180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:27.683363 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:27.683412 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:27.683413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.684160 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.684161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:27.684678 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:27.684679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.685046 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.685047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:27.685101 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:27.685673 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:27.685677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:27.687428 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
W20260810 04:46:27.687600 31341 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:27.682924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:47877 (local address 127.14.237.62:38689)
0810 04:46:27.683027 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:27.683029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:27.683052 (+ 23us) server_negotiation.cc:407] Connection header received
0810 04:46:27.683113 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:27.683114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:27.683126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:27.683175 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:27.683474 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.684058 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.684760 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:27.684975 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:27.685044 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:27.687520 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:47877: BlockingRecv error: recv got EOF from 127.14.237.3:47877 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 04:46:28 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:28 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:28.445525 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:28.440708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:59735)
0810 04:46:28.440870 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:28.440884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:28.440949 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:28.441107 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:28.441108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:28.441302 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:28.441355 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:28.441356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.442054 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.442056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:28.442617 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:28.442619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.443005 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.443006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:28.443065 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:28.443533 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:28.443537 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:28.445464 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
W20260810 04:46:28.445724 31347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:28.440782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:59735 (local address 127.14.237.62:38689)
0810 04:46:28.440939 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:28.440941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:28.440951 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:28.441002 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:28.441003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:28.441015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:28.441095 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:28.441431 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.441940 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.442712 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.442917 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.443004 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:28.445593 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:59735: BlockingRecv error: recv got EOF from 127.14.237.1:59735 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
Aug 10 04:46:28 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:28 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:28.570300 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:28.564963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:53001)
0810 04:46:28.565162 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:28.565177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:28.565323 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:28.565505 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:28.565507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:28.565723 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:28.565799 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:28.565803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.566788 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.566789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:28.567321 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:28.567323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.567674 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.567674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:28.567717 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:28.568258 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:28.568263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:28.570237 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
W20260810 04:46:28.570492 31347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:28.565071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:53001 (local address 127.14.237.62:38689)
0810 04:46:28.565166 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:28.565169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:28.565280 (+ 111us) server_negotiation.cc:407] Connection header received
0810 04:46:28.565399 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:28.565400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:28.565415 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:28.565498 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:28.565914 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.566713 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.567401 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.567603 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.567653 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:28.570362 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:53001: BlockingRecv error: recv got EOF from 127.14.237.2:53001 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 04:46:28 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:28 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:28.692875 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:28.688035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:37889)
0810 04:46:28.688233 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:28.688253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:28.688320 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:28.688494 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:28.688496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:28.688683 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:28.688738 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:28.688739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.689426 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.689426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:28.689958 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:28.689960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.690280 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.690281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:28.690324 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:28.690811 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:28.690816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:28.692776 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":111,"threads_started":1}
W20260810 04:46:28.692929 31347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:28.688092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:37889 (local address 127.14.237.62:38689)
0810 04:46:28.688157 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:28.688159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:28.688318 (+ 159us) server_negotiation.cc:407] Connection header received
0810 04:46:28.688379 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:28.688380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:28.688392 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:28.688461 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:28.688803 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.689354 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.690032 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:28.690205 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:28.690261 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:28.692863 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:37889: BlockingRecv error: recv got EOF from 127.14.237.3:37889 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 04:46:29 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:29 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:29.450835 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:29.445995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:35335)
0810 04:46:29.446126 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:29.446136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:29.446184 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:29.446368 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:29.446369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:29.446536 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:29.446587 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:29.446588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.447226 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.447227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:29.447713 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:29.447714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.448121 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.448121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:29.448160 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:29.448638 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:29.448642 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:29.450733 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
W20260810 04:46:29.450896 31358 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:29.446061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:35335 (local address 127.14.237.62:38689)
0810 04:46:29.446224 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:29.446225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:29.446234 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:29.446270 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:29.446270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:29.446283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:29.446357 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:29.446643 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.447153 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.447823 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.448018 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.448113 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:29.450818 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:35335: BlockingRecv error: recv got EOF from 127.14.237.1:35335 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
Aug 10 04:46:29 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:29 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:29.576054 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:29.570900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:47017)
0810 04:46:29.571092 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:29.571124 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:29.571210 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:29.571387 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:29.571388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:29.571616 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:29.571664 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:29.571666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.572345 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.572346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:29.572880 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:29.572882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.573366 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.573367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:29.573423 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:29.573872 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:29.573877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:29.575995 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":104,"threads_started":1}
W20260810 04:46:29.576188 31358 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:29.570995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:47017 (local address 127.14.237.62:38689)
0810 04:46:29.571077 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:29.571079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:29.571189 (+ 110us) server_negotiation.cc:407] Connection header received
0810 04:46:29.571289 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:29.571290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:29.571304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:29.571380 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:29.571749 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.572256 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.572968 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.573178 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.573278 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:29.576085 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:47017: BlockingRecv error: recv got EOF from 127.14.237.2:47017 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 04:46:29 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:29 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:29.700904 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:29.693414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:52243)
0810 04:46:29.693565 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:29.693577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:29.693635 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:29.693767 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:29.693768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:29.693937 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:29.693988 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:29.693989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.694740 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.694741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:29.695424 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:29.695427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.695741 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.695742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:29.695780 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:29.699015 (+ 3235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:29.699020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:29.700846 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
W20260810 04:46:29.701038 31358 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:29.693471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:52243 (local address 127.14.237.62:38689)
0810 04:46:29.693546 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:29.693548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:29.693636 (+ 88us) server_negotiation.cc:407] Connection header received
0810 04:46:29.693679 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:29.693680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:29.693693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:29.693748 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:29.694050 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.694641 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.695498 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:29.695677 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:29.695733 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:29.700952 (+ 5219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:52243: BlockingRecv error: recv got EOF from 127.14.237.3:52243 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 04:46:30 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:30 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:30.456305 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:30.451374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:32813)
0810 04:46:30.451560 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:30.451571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:30.451629 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:30.451777 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:30.451777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:30.451956 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:30.452008 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:30.452010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.452712 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.452712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:30.453298 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:30.453300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.453653 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.453654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:30.453693 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:30.454184 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:30.454188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:30.456245 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":118,"threads_started":1}
W20260810 04:46:30.456445 31362 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:30.451477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:32813 (local address 127.14.237.62:38689)
0810 04:46:30.451617 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:30.451620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:30.451630 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:30.451690 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:30.451691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:30.451703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:30.451759 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:30.452081 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.452641 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.453372 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.453583 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.453646 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:30.456349 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:32813: BlockingRecv error: recv got EOF from 127.14.237.1:32813 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
Aug 10 04:46:30 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:30 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:30.581152 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:30.576537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:42697)
0810 04:46:30.576725 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:30.576735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:30.576795 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:30.576961 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:30.576964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:30.577141 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:30.577270 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:30.577276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.577972 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.577973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:30.578528 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:30.578531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.578875 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.578876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:30.578913 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:30.579354 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:30.579358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:30.581080 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
W20260810 04:46:30.581284 31362 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:30.576649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:42697 (local address 127.14.237.62:38689)
0810 04:46:30.576710 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:30.576712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:30.576797 (+ 85us) server_negotiation.cc:407] Connection header received
0810 04:46:30.576848 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:30.576848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:30.576859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:30.576906 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:30.577348 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.577901 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.578606 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.578796 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.578861 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:30.581175 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:42697: BlockingRecv error: recv got EOF from 127.14.237.2:42697 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 04:46:30 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:30 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:30.706301 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:30.701520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:57659)
0810 04:46:30.701687 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:30.701697 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:30.701769 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:30.701904 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:30.701905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:30.702073 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:30.702122 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:30.702123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.702762 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.702762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:30.703264 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:30.703265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.703599 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.703600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:30.703636 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:30.704099 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:30.704103 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:30.706225 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
W20260810 04:46:30.706408 31362 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:30.701608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:57659 (local address 127.14.237.62:38689)
0810 04:46:30.701686 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:30.701688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:30.701768 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:46:30.701814 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:30.701815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:30.701827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:30.701881 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:30.702187 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.702687 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.703344 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:30.703530 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:30.703586 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:30.706335 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:57659: BlockingRecv error: recv got EOF from 127.14.237.3:57659 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 04:46:31 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:31 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:31.461781 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:31.456853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:48431)
0810 04:46:31.457045 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:31.457057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:31.457118 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:31.457282 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:31.457283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:31.457472 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:31.457524 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:31.457525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.458164 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.458165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:31.458736 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:31.458738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.459087 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.459088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:31.459132 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:31.459697 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:31.459703 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:31.461677 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":114,"threads_started":1}
W20260810 04:46:31.461830 31373 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:31.456975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:48431 (local address 127.14.237.62:38689)
0810 04:46:31.457133 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:31.457134 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:31.457143 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:31.457167 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:31.457168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:31.457179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:31.457275 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:31.457595 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.458069 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.458816 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.459018 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.459067 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:31.461761 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:48431: BlockingRecv error: recv got EOF from 127.14.237.1:48431 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
Aug 10 04:46:31 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:31 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:31.586361 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:31.581634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:58417)
0810 04:46:31.581778 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:31.581795 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:31.581851 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:31.581984 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:31.581985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:31.582148 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:31.582196 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:31.582197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.582807 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.582807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:31.583489 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:31.583494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.583890 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.583891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:31.583941 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:31.584417 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:31.584421 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:31.586266 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
W20260810 04:46:31.586411 31373 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:31.581717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:58417 (local address 127.14.237.62:38689)
0810 04:46:31.581788 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:31.581790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:31.581850 (+ 60us) server_negotiation.cc:407] Connection header received
0810 04:46:31.581897 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:31.581898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:31.581908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:31.581962 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:31.582265 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.582740 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.583566 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.583816 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.583891 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:31.586348 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:58417: BlockingRecv error: recv got EOF from 127.14.237.2:58417 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 04:46:31 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:31 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:31.712090 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:31.706858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:57253)
0810 04:46:31.707055 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:31.707068 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:31.707165 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:31.707339 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:31.707340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:31.707555 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:31.707604 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:31.707606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.708290 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.708291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:31.708823 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:31.708825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.709178 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.709179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:31.709276 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:31.709848 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:31.709854 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:31.711991 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
W20260810 04:46:31.712131 31373 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:31.706976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:57253 (local address 127.14.237.62:38689)
0810 04:46:31.707054 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:31.707056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:31.707145 (+ 89us) server_negotiation.cc:407] Connection header received
0810 04:46:31.707244 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:31.707245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:31.707265 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:31.707332 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:31.707696 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.708188 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.708916 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:31.709102 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:31.709180 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:31.712068 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:57253: BlockingRecv error: recv got EOF from 127.14.237.3:57253 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 04:46:32 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:32 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:32.466897 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:32.462278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:56241)
0810 04:46:32.462404 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:32.462417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:32.462486 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:32.462621 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:32.462622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:32.462790 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:32.462840 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:32.462842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.463552 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.463553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:32.464082 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:32.464084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.464450 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.464450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:32.464490 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:32.464955 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:32.464959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:32.466824 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260810 04:46:32.467034 31379 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:32.462351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:56241 (local address 127.14.237.62:38689)
0810 04:46:32.462493 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:32.462495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:32.462505 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:32.462530 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:32.462530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:32.462542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:32.462606 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:32.462907 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.463483 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.464178 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.464376 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.464442 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:32.466921 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:56241: BlockingRecv error: recv got EOF from 127.14.237.1:56241 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
Aug 10 04:46:32 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:32 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:32.591674 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:32.586860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:35749)
0810 04:46:32.587000 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:32.587010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:32.587059 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:32.587265 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:32.587266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:32.587447 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:32.587496 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:32.587498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.588165 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.588166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:32.588776 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:32.588778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.589188 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.589189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:32.589277 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:32.589764 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:32.589769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:32.591565 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20260810 04:46:32.591718 31379 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:32.587041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:35749 (local address 127.14.237.62:38689)
0810 04:46:32.587118 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:32.587120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:32.587127 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:46:32.587160 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:32.587161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:32.587172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:32.587253 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:32.587579 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.588073 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.588862 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.589122 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.589190 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:32.591653 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:35749: BlockingRecv error: recv got EOF from 127.14.237.2:35749 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 04:46:32 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:32 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:32.717733 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:32.712618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:48105)
0810 04:46:32.712914 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:32.712928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:32.713020 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:32.713193 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:32.713194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:32.713418 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:32.713466 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:32.713468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.714169 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.714170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:32.714687 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:32.714688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.715054 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.715055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:32.715127 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:32.715573 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:32.715578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:32.717623 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
W20260810 04:46:32.717772 31379 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:32.712741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:48105 (local address 127.14.237.62:38689)
0810 04:46:32.712917 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:32.712920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:32.712998 (+ 78us) server_negotiation.cc:407] Connection header received
0810 04:46:32.713114 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:32.713115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:32.713131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:32.713178 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:32.713548 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.714048 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.714772 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:32.714976 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:32.715056 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:32.717706 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:48105: BlockingRecv error: recv got EOF from 127.14.237.3:48105 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Aug 10 04:46:33 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:33 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:33.472246 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:33.467362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:42941)
0810 04:46:33.467520 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:33.467551 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:33.467612 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:33.467807 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:33.467808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:33.468073 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:33.468158 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:33.468160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.468966 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.468966 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:33.469519 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:33.469521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.469844 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.469845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:33.469883 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:33.470315 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:33.470319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:33.472186 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
W20260810 04:46:33.472402 31390 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:33.467522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:42941 (local address 127.14.237.62:38689)
0810 04:46:33.467639 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:33.467641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:33.467649 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:46:33.467683 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:33.467683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:33.467694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:33.467783 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:33.468271 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.468889 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.469590 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.469781 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.469832 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:33.472307 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:42941: BlockingRecv error: recv got EOF from 127.14.237.1:42941 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
Aug 10 04:46:33 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:33 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:33.596989 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:33.592298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:54619)
0810 04:46:33.592446 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:33.592457 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:33.592517 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:33.592653 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:33.592654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:33.592826 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:33.592874 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:33.592876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.593660 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.593660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:33.594191 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:33.594192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.594498 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.594499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:33.594538 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:33.595021 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:33.595025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:33.596895 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
W20260810 04:46:33.597061 31390 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:33.592366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:54619 (local address 127.14.237.62:38689)
0810 04:46:33.592446 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:33.592448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:33.592518 (+ 70us) server_negotiation.cc:407] Connection header received
0810 04:46:33.592579 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:33.592579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:33.592590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:33.592635 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:33.592939 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.593594 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.594265 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.594437 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.594485 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:33.596996 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:54619: BlockingRecv error: recv got EOF from 127.14.237.2:54619 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 04:46:33 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:33 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:33.722772 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:33.718252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:39187)
0810 04:46:33.718407 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:33.718417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:33.718471 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:33.718616 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:33.718617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:33.718770 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:33.718819 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:33.718820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.719395 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.719395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:33.719897 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:33.719898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.720257 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.720258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:33.720296 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:33.720828 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:33.720833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:33.722703 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
W20260810 04:46:33.722886 31390 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:33.718313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:39187 (local address 127.14.237.62:38689)
0810 04:46:33.718356 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:33.718358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:33.718456 (+ 98us) server_negotiation.cc:407] Connection header received
0810 04:46:33.718510 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:33.718511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:33.718539 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:33.718616 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:33.718866 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.719327 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.719962 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:33.720194 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:33.720248 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:33.722789 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:39187: BlockingRecv error: recv got EOF from 127.14.237.3:39187 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 10 04:46:34 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:34 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:34.479712 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:34.472745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:53065)
0810 04:46:34.472902 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:34.472917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:34.472988 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:34.473161 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:34.473162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:34.473407 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:34.473473 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:34.473476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.474386 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.474387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:34.475144 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:34.475146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.475612 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.475613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:34.475661 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:34.476230 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:34.476236 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:34.479601 (+ 3365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20260810 04:46:34.479847 31399 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:34.472837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:53065 (local address 127.14.237.62:38689)
0810 04:46:34.472952 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:34.472954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:34.472986 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:46:34.473063 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:34.473064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:34.473080 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:34.473144 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:34.473583 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.474293 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.475235 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.475510 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.475561 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:34.479744 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:53065: BlockingRecv error: recv got EOF from 127.14.237.1:53065 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
Aug 10 04:46:34 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:34 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:34.602196 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:34.597588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:54091)
0810 04:46:34.597736 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:34.597750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:34.597808 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:34.597959 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:34.597960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:34.598137 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:34.598187 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:34.598189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.598790 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.598791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:34.599280 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:34.599282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.599670 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.599671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:34.599765 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:34.600342 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:34.600348 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:34.602128 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
W20260810 04:46:34.602324 31399 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:34.597705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:54091 (local address 127.14.237.62:38689)
0810 04:46:34.597789 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:34.597791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:34.597806 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:46:34.597870 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:34.597871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:34.597883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:34.597941 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:34.598252 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.598725 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.599347 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.599514 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.599562 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:34.602242 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:54091: BlockingRecv error: recv got EOF from 127.14.237.2:54091 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 04:46:34 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:34 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:34.731379 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:34.723377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:60487)
0810 04:46:34.723535 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:34.723544 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:34.723644 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:34.723778 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:34.723778 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:34.723948 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:34.724018 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:34.724019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.724686 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.724687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:34.725197 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:34.725198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.725624 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.725625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:34.725689 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:34.729060 (+ 3371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:34.729065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:34.731302 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
W20260810 04:46:34.731505 31399 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:34.723451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:60487 (local address 127.14.237.62:38689)
0810 04:46:34.723555 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:34.723557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:34.723629 (+ 72us) server_negotiation.cc:407] Connection header received
0810 04:46:34.723693 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:34.723693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:34.723706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:34.723760 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:34.724112 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.724619 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.725333 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:34.725530 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:34.725620 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:34.731404 (+ 5784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:60487: BlockingRecv error: recv got EOF from 127.14.237.3:60487 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 04:46:35 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:35 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:35.485538 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:35.480509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:35565)
0810 04:46:35.480651 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:35.480661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:35.480718 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:35.480873 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:35.480874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:35.481043 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:35.481092 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:35.481093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.481941 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.481942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:35.482584 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:35.482587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.482951 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.482952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:35.482991 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:35.483554 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:35.483558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:35.485437 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
W20260810 04:46:35.485598 31404 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:35.480568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:35565 (local address 127.14.237.62:38689)
0810 04:46:35.480717 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:35.480719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:35.480731 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:46:35.480776 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:35.480777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:35.480789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:35.480865 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:35.481179 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.481818 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.482665 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.482877 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.482941 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:35.485521 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:35565: BlockingRecv error: recv got EOF from 127.14.237.1:35565 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
Aug 10 04:46:35 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:35 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:35.607323 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:35.602752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:59573)
0810 04:46:35.602929 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:35.602941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:35.603029 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:35.603224 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:35.603224 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:35.603394 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:35.603442 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:35.603443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.604105 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.604105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:35.604605 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:35.604606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.604937 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.604937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:35.604975 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:35.605457 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:35.605461 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:35.607266 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
W20260810 04:46:35.607460 31404 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:35.602840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:59573 (local address 127.14.237.62:38689)
0810 04:46:35.602899 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:35.602901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:35.603108 (+ 207us) server_negotiation.cc:407] Connection header received
0810 04:46:35.603142 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:35.603142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:35.603158 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:35.603208 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:35.603509 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.604029 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.604682 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.604869 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.604927 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:35.607373 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:59573: BlockingRecv error: recv got EOF from 127.14.237.2:59573 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 10 04:46:35 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:35 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:35.736712 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:35.731979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:55935)
0810 04:46:35.732178 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:35.732190 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:35.732247 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:35.732416 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:35.732417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:35.732669 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:35.732733 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:35.732737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.733405 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.733406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:35.733981 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:35.733983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.734284 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.734285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:35.734322 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:35.734812 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:35.734820 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:35.736655 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260810 04:46:35.736850 31404 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:35.732129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:55935 (local address 127.14.237.62:38689)
0810 04:46:35.732211 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:35.732213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:35.732250 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:46:35.732313 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:35.732314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:35.732330 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:35.732388 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:35.732826 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.733334 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.734052 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:35.734222 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:35.734273 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:35.736763 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:55935: BlockingRecv error: recv got EOF from 127.14.237.3:55935 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 04:46:36 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:36 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:36.490828 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:36.486126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:60351)
0810 04:46:36.486279 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:36.486289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:36.486341 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:36.486493 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:36.486494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:36.486678 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:36.486727 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:36.486729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.487359 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.487360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:36.487901 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:36.487902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.488225 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.488226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:36.488268 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:36.488858 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:36.488863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:36.490758 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
W20260810 04:46:36.490938 31410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:36.486177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:60351 (local address 127.14.237.62:38689)
0810 04:46:36.486314 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:36.486316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:36.486326 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:36.486402 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:36.486402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:36.486414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:36.486477 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:36.486794 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.487293 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.487974 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.488161 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.488212 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:36.490856 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:60351: BlockingRecv error: recv got EOF from 127.14.237.1:60351 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
Aug 10 04:46:36 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:36 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:36.613072 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:36.607813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:36731)
0810 04:46:36.607998 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:36.608012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:36.608082 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:36.608270 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:36.608271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:36.608487 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:36.608531 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:36.608533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.609270 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.609271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:36.609848 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:36.609850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.610254 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.610254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:36.610320 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:36.610884 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:36.610889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:36.613003 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
W20260810 04:46:36.613180 31410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:36.607959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:36731 (local address 127.14.237.62:38689)
0810 04:46:36.608036 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:36.608038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:36.608062 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:46:36.608190 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:36.608191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:36.608205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:36.608254 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:36.608623 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.609146 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.609943 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.610173 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.610253 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:36.613099 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:36731: BlockingRecv error: recv got EOF from 127.14.237.2:36731 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 04:46:36 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:36 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:36.742018 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:36.737203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:43025)
0810 04:46:36.737377 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:36.737389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:36.737456 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:36.737588 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:36.737589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:36.737822 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:36.737871 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:36.737872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.738486 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.738487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:36.739052 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:36.739054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.739382 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.739383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:36.739428 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:36.739970 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:36.739975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:36.741949 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":124,"threads_started":1}
W20260810 04:46:36.742138 31410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:36.737303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:43025 (local address 127.14.237.62:38689)
0810 04:46:36.737374 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:36.737376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:36.737456 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:46:36.737500 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:36.737500 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:36.737513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:36.737564 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:36.737938 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.738415 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.739135 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:36.739309 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:36.739358 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:36.742037 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:43025: BlockingRecv error: recv got EOF from 127.14.237.3:43025 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 04:46:37 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:37 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:37.495923 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:37.491318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:54081)
0810 04:46:37.491439 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:37.491453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:37.491519 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:37.491660 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:37.491660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:37.491878 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:37.491942 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:37.491944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.492545 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.492546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:37.493088 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:37.493090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.493453 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.493453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:37.493508 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:37.494051 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:37.494056 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:37.495866 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
W20260810 04:46:37.496068 31414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:37.491403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:54081 (local address 127.14.237.62:38689)
0810 04:46:37.491559 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:37.491561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:37.491568 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:46:37.491590 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:37.491591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:37.491601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:37.491655 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:37.492005 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.492479 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.493166 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.493390 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.493442 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:37.495976 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:54081: BlockingRecv error: recv got EOF from 127.14.237.1:54081 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
Aug 10 04:46:37 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:37 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:37.618486 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:37.613696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:60039)
0810 04:46:37.613849 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:37.613865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:37.613922 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:37.614075 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:37.614076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:37.614257 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:37.614303 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:37.614304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.614929 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.614929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:37.615453 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:37.615455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.615782 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.615782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:37.615821 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:37.616306 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:37.616310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:37.618400 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
W20260810 04:46:37.618623 31414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:37.613779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:60039 (local address 127.14.237.62:38689)
0810 04:46:37.613838 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:37.613840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:37.613920 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:46:37.614003 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:37.614004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:37.614015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:37.614058 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:37.614369 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.614857 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.615527 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.615716 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.615767 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:37.618480 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:60039: BlockingRecv error: recv got EOF from 127.14.237.2:60039 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 10 04:46:37 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:37 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:37.747287 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:37.742504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:44789)
0810 04:46:37.742661 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:37.742672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:37.742730 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:37.742866 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:37.742867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:37.743041 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:37.743102 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:37.743103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.743837 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.743838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:37.744410 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:37.744412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.744741 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.744742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:37.744794 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:37.745313 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:37.745318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:37.747230 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260810 04:46:37.747422 31414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:37.742595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:44789 (local address 127.14.237.62:38689)
0810 04:46:37.742646 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:37.742647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:37.742727 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:46:37.742783 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:37.742784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:37.742796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:37.742848 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:37.743192 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.743759 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.744483 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:37.744677 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:37.744729 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:37.747311 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:44789: BlockingRecv error: recv got EOF from 127.14.237.3:44789 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 10 04:46:38 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:38 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:38.500766 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:38.496406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:51649)
0810 04:46:38.496591 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:38.496602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:38.496661 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:38.496813 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:38.496813 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:38.496993 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:38.497044 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:38.497045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.497700 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.497700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:38.498201 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:38.498202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.498534 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.498535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:38.498576 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:38.498993 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:38.498998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:38.500704 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
W20260810 04:46:38.500877 31423 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:38.496453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:51649 (local address 127.14.237.62:38689)
0810 04:46:38.496602 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:38.496604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:38.496645 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:46:38.496722 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:38.496723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:38.496734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:38.496790 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:38.497113 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.497638 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.498284 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.498468 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.498519 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:38.500798 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:51649: BlockingRecv error: recv got EOF from 127.14.237.1:51649 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
Aug 10 04:46:38 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:38 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:38.623929 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:38.619013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:44129)
0810 04:46:38.619235 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:38.619247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:38.619331 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:38.619475 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:38.619475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:38.619657 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:38.619706 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:38.619708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.620391 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.620392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:38.620911 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:38.620912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.621388 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.621389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:38.621458 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:38.621898 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:38.621902 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:38.623871 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":115,"threads_started":1}
W20260810 04:46:38.624069 31423 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:38.619176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:44129 (local address 127.14.237.62:38689)
0810 04:46:38.619240 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:38.619242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:38.619318 (+ 76us) server_negotiation.cc:407] Connection header received
0810 04:46:38.619391 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:38.619392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:38.619403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:38.619456 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:38.619787 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.620315 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.621028 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.621289 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.621386 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:38.623978 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:44129: BlockingRecv error: recv got EOF from 127.14.237.2:44129 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 04:46:38 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:38 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:38.752731 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:38.747854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:56271)
0810 04:46:38.748034 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:38.748048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:38.748104 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:38.748271 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:38.748272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:38.748465 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:38.748513 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:38.748514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.749268 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.749269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:38.749808 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:38.749810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.750178 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.750178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:38.750251 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:38.750763 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:38.750768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:38.752627 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
W20260810 04:46:38.752912 31423 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:38.747993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:56271 (local address 127.14.237.62:38689)
0810 04:46:38.748074 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:38.748076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:38.748101 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:46:38.748166 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:38.748167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:38.748184 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:38.748261 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:38.748599 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.749111 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.749913 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:38.750102 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:38.750178 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:38.752753 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:56271: BlockingRecv error: recv got EOF from 127.14.237.3:56271 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 04:46:39 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:39 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:39.506217 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:39.501395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:45351)
0810 04:46:39.501534 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:39.501547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:39.501601 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:39.501753 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:39.501754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:39.501922 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:39.501972 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:39.501973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.502628 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.502629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:39.503118 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:39.503119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.503431 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.503431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:39.503471 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:39.504026 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:39.504034 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:39.506158 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260810 04:46:39.506367 31428 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:39.501479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:45351 (local address 127.14.237.62:38689)
0810 04:46:39.501622 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:39.501624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:39.501633 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:39.501662 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:39.501663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:39.501673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:39.501734 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:39.502044 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.502561 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.503186 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.503367 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.503417 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:39.506275 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:45351: BlockingRecv error: recv got EOF from 127.14.237.1:45351 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
Aug 10 04:46:39 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:39 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:39.629482 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:39.624436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:43437)
0810 04:46:39.624614 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:39.624627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:39.624712 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:39.624860 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:39.624861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:39.625045 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:39.625090 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:39.625092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.626093 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.626098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:39.626641 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:39.626643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.627001 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.627001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:39.627058 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:39.627499 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:39.627504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:39.629413 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
W20260810 04:46:39.629637 31428 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:39.624525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:43437 (local address 127.14.237.62:38689)
0810 04:46:39.624605 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:39.624608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:39.624694 (+ 86us) server_negotiation.cc:407] Connection header received
0810 04:46:39.624770 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:39.624771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:39.624783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:39.624856 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:39.625454 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.625958 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.626734 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.626926 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.627004 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:39.629563 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:43437: BlockingRecv error: recv got EOF from 127.14.237.2:43437 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 04:46:39 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:39 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.3@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:39.760906 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:39.753180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.3:37591)
0810 04:46:39.753354 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:39.753365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:39.753425 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:39.753548 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:39.753549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:39.753721 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:39.753769 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:39.753771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.754391 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.754392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:39.754897 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:39.754898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.755200 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.755201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:39.755241 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:39.758901 (+ 3660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:39.758905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:39.760847 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
W20260810 04:46:39.761113 31428 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:39.753273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:37591 (local address 127.14.237.62:38689)
0810 04:46:39.753338 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:39.753340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:39.753426 (+ 86us) server_negotiation.cc:407] Connection header received
0810 04:46:39.753468 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:39.753468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:39.753481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:39.753530 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:39.753830 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.754321 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.754966 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:39.755137 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:39.755184 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:39.761037 (+ 5853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.3:37591: BlockingRecv error: recv got EOF from 127.14.237.3:37591 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 04:46:40 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:40 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.1@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:40.511513 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:40.506791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.1:40909)
0810 04:46:40.506972 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:40.506983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:40.507033 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:40.507194 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:40.507195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:40.507398 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:40.507444 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:40.507445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.508047 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.508047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:40.508547 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:40.508549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.508947 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.508949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:40.509019 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:40.509563 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:40.509569 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:40.511456 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
W20260810 04:46:40.511610 31434 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:40.506910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:40909 (local address 127.14.237.62:38689)
0810 04:46:40.507033 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:40.507035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:40.507076 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:46:40.507099 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:40.507100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:40.507120 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:40.507170 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:40.507515 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.507984 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.508659 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.508855 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.508927 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:40.511540 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.1:40909: BlockingRecv error: recv got EOF from 127.14.237.1:40909 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
Aug 10 04:46:40 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:46:40 dist-test-slave-8hhm krb5kdc[29649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.237.2@KRBTEST.COM for oryx/127.14.237.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:46:40.634593 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:40.630006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38689 (local address 127.14.237.2:52641)
0810 04:46:40.630184 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:40.630194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:40.630260 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:40.630392 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:40.630392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:40.630556 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:40.630606 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:40.630607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.631196 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.631214 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:40.631731 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:40.631733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.632131 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.632131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:40.632171 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:40.632697 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:40.632702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:40.634532 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.237.62:38689: Server oryx/127.14.237.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
W20260810 04:46:40.634718 31434 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:40.630088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:52641 (local address 127.14.237.62:38689)
0810 04:46:40.630156 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:40.630158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:40.630284 (+ 126us) server_negotiation.cc:407] Connection header received
0810 04:46:40.630318 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:40.630318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:40.630332 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:40.630388 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:40.630667 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.631132 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.631855 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.632051 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.632124 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:40.634636 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.237.2:52641: BlockingRecv error: recv got EOF from 127.14.237.2:52641 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:46:40.671783 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 30651
I20260810 04:46:40.687968 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 30893
I20260810 04:46:40.712141 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 31118
I20260810 04:46:40.723953 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 30535
2026-08-10T04:46:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17815 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T04:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:40Z Disabled control of system clock
I20260810 04:46:40.758431 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:34607
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:34607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 04:46:40.841537 31443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:40.841739 31443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:40.841759 31443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:40.842994 31443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:40.843036 31443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:40.843050 31443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:40.843103 31443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:40.843129 31443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:40.844535 31443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:34607
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:34607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:40.844790 31443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:40.845105 31443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:46:40.850301 31443 server_base.cc:1034] running on GCE node
W20260810 04:46:40.850268 31449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:40.850282 31454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:40.850309 31451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:40.850751 31443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:40.851069 31443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:40.852253 31443 hybrid_clock.cc:648] HybridClock initialized: now 1786337200852237 us; error 28 us; skew 500 ppm
I20260810 04:46:40.853626 31443 webserver.cc:460] Webserver started at http://127.14.237.62:39583/ using document root <none> and password file <none>
I20260810 04:46:40.853920 31443 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:40.853991 31443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:40.854153 31443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:40.855248 31443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "833b1cf1488849b7b97f8c66ecc5fa94"
format_stamp: "Formatted at 2026-08-10 04:46:40 on dist-test-slave-8hhm"
server_key: "cd23be987481fce248476c08cf897cb7"
server_key_iv: "51e53ca47c7fc0605d836698f42d9a1a"
server_key_version: "encryptionkey@0"
I20260810 04:46:40.855638 31443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "833b1cf1488849b7b97f8c66ecc5fa94"
format_stamp: "Formatted at 2026-08-10 04:46:40 on dist-test-slave-8hhm"
server_key: "cd23be987481fce248476c08cf897cb7"
server_key_iv: "51e53ca47c7fc0605d836698f42d9a1a"
server_key_version: "encryptionkey@0"
I20260810 04:46:40.856949 31443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:40.857718 31463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:40.857880 31443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:40.857973 31443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "833b1cf1488849b7b97f8c66ecc5fa94"
format_stamp: "Formatted at 2026-08-10 04:46:40 on dist-test-slave-8hhm"
server_key: "cd23be987481fce248476c08cf897cb7"
server_key_iv: "51e53ca47c7fc0605d836698f42d9a1a"
server_key_version: "encryptionkey@0"
I20260810 04:46:40.858058 31443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:40.887467 31443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:40.887864 31443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:40.888149 31443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:40.892869 31443 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:34607
I20260810 04:46:40.892904 31546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:34607 every 8 connection(s)
I20260810 04:46:40.893371 31443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:40.894069 31547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:40.896731 31547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94: Bootstrap starting.
I20260810 04:46:40.897408 31547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:40.897709 31547 log.cc:824] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:40.898432 31547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94: No bootstrap required, opened a new log
I20260810 04:46:40.899962 31547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833b1cf1488849b7b97f8c66ecc5fa94" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34607 } }
I20260810 04:46:40.900097 31547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:40.900120 31547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833b1cf1488849b7b97f8c66ecc5fa94, State: Initialized, Role: FOLLOWER
I20260810 04:46:40.900290 31547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833b1cf1488849b7b97f8c66ecc5fa94" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34607 } }
I20260810 04:46:40.900353 31547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:40.900394 31547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:40.900441 31547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:40.900969 31547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833b1cf1488849b7b97f8c66ecc5fa94" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34607 } }
I20260810 04:46:40.901086 31547 leader_election.cc:302] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 833b1cf1488849b7b97f8c66ecc5fa94; no voters:
I20260810 04:46:40.901293 31547 leader_election.cc:288] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:40.901329 31551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:40.901491 31551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [term 1 LEADER]: Becoming Leader. State: Replica: 833b1cf1488849b7b97f8c66ecc5fa94, State: Running, Role: LEADER
I20260810 04:46:40.901558 31547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:40.901649 31551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833b1cf1488849b7b97f8c66ecc5fa94" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34607 } }
I20260810 04:46:40.902100 31553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833b1cf1488849b7b97f8c66ecc5fa94. Latest consensus state: current_term: 1 leader_uuid: "833b1cf1488849b7b97f8c66ecc5fa94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833b1cf1488849b7b97f8c66ecc5fa94" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34607 } } }
I20260810 04:46:40.902235 31552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "833b1cf1488849b7b97f8c66ecc5fa94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833b1cf1488849b7b97f8c66ecc5fa94" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 34607 } } }
I20260810 04:46:40.902267 31553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:40.902740 31552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:40.902794 31561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:40.903276 31561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:40.903542 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 31443
I20260810 04:46:40.903649 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:40.903867 15284 external_mini_cluster.cc:1428] Setting key e70994b25eabd6c8626d4622e5a3569d
I20260810 04:46:40.905134 31561 catalog_manager.cc:1347] Generated new cluster ID: 1717ca2d660541a28fca0aa4a59be011
I20260810 04:46:40.905189 31561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:40.910261 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:40.904530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.14.237.62:34607)
0810 04:46:40.904712 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:40.904721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:40.904755 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:46:40.904858 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:40.904864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:40.904915 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:40.905137 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:40.905519 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.906904 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.908008 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:40.908659 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:40.908793 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:40.909529 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:40.909533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:40.909534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:40.909749 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:40.909788 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:40.909803 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:40.909893 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:40.909960 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 04:46:40.910041 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260810 04:46:40.914498 31561 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:40.914991 31561 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:40.927871 31561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 833b1cf1488849b7b97f8c66ecc5fa94: Generated new TSK 0
I20260810 04:46:40.928056 31561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:46:40.934026 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34607
--builtin_ntp_servers=127.14.237.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 04:46:41.019047 31581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:41.019245 31581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:41.019265 31581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:41.020404 31581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:41.020444 31581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:41.020468 31581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:41.021781 31581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:41.022045 31581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:41.022373 31581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:41.028198 31594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.028173 31588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.028177 31592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.028553 31581 server_base.cc:1034] running on GCE node
I20260810 04:46:41.028749 31581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:41.029014 31581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:41.030187 31581 hybrid_clock.cc:648] HybridClock initialized: now 1786337201030160 us; error 54 us; skew 500 ppm
I20260810 04:46:41.031378 31581 webserver.cc:460] Webserver started at http://127.14.237.1:42357/ using document root <none> and password file <none>
I20260810 04:46:41.031680 31581 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:41.031751 31581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:41.031899 31581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:41.032963 31581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "bfb9da28bb8c43c6938bc8df5abc34e1"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "7e2beb81ff8def864d89bf857a3b5df8"
server_key_iv: "d3af2e92ec4f38a5b75f0a334d34ae8e"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.033396 31581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "bfb9da28bb8c43c6938bc8df5abc34e1"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "7e2beb81ff8def864d89bf857a3b5df8"
server_key_iv: "d3af2e92ec4f38a5b75f0a334d34ae8e"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.035008 31581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:46:41.035785 31607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.035967 31581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:41.036069 31581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "bfb9da28bb8c43c6938bc8df5abc34e1"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "7e2beb81ff8def864d89bf857a3b5df8"
server_key_iv: "d3af2e92ec4f38a5b75f0a334d34ae8e"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.036183 31581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:41.044251 31581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:41.044523 31581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:41.044667 31581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:41.044874 31581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:41.045195 31581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:41.045300 31581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.045342 31581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:41.045370 31581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.051498 31581 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:45733
I20260810 04:46:41.051545 31804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:45733 every 8 connection(s)
I20260810 04:46:41.051906 31581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:41.057268 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.052604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:46643 (local address 127.14.237.62:34607)
0810 04:46:41.052670 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.052672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.052915 (+ 243us) server_negotiation.cc:407] Connection header received
0810 04:46:41.053523 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.053524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.053540 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.053592 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.054325 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.054834 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.055840 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.056050 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.056129 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:41.056942 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:41.056944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:41.056944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:41.056949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:41.056964 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:41.056968 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:41.057031 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:41.057132 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 04:46:41.057162 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:46:41.057490 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.052610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34607 (local address 127.14.237.1:46643)
0810 04:46:41.052799 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:41.052833 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:41.053408 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:41.053665 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:41.053669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:41.053677 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:41.054212 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:41.054218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.054949 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.054950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:41.055743 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:41.055745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.056138 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.056139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:41.056279 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:41.056799 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:41.056800 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:41.056828 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:41.056851 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:41.057037 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:41.057042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:41.057045 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:41.057121 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:46:41.057264 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260810 04:46:41.058035 31808 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34607
I20260810 04:46:41.058116 31808 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:41.058336 31808 heartbeater.cc:511] Master 127.14.237.62:34607 requested a full tablet report, sending...
I20260810 04:46:41.058882 31488 ts_manager.cc:194] Registered new tserver with Master: bfb9da28bb8c43c6938bc8df5abc34e1 (127.14.237.1:45733)
I20260810 04:46:41.059479 31488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.1:46643
I20260810 04:46:41.060087 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 31581
I20260810 04:46:41.060180 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:41.060405 15284 external_mini_cluster.cc:1428] Setting key 5401c1abd5a7c5ac67a395af501177d2
I20260810 04:46:41.061435 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34607
--builtin_ntp_servers=127.14.237.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 04:46:41.143419 31814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:41.143658 31814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:41.143723 31814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:41.144826 31814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:41.144889 31814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:41.144927 31814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:41.146268 31814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:41.146533 31814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:41.146845 31814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:41.151842 31823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.151814 31831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.151914 31814 server_base.cc:1034] running on GCE node
W20260810 04:46:41.151834 31824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.152263 31814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:41.152541 31814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:41.153705 31814 hybrid_clock.cc:648] HybridClock initialized: now 1786337201153680 us; error 41 us; skew 500 ppm
I20260810 04:46:41.154940 31814 webserver.cc:460] Webserver started at http://127.14.237.2:33313/ using document root <none> and password file <none>
I20260810 04:46:41.155174 31814 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:41.155260 31814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:41.155422 31814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:41.156517 31814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "7abb46fb6bad4d91b1945ec2b7c3992a"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "8940d67b9de0d1110ff637537b803f83"
server_key_iv: "808db41976afe5376468db39f21bd5ff"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.156893 31814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "7abb46fb6bad4d91b1945ec2b7c3992a"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "8940d67b9de0d1110ff637537b803f83"
server_key_iv: "808db41976afe5376468db39f21bd5ff"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.158345 31814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:41.159066 31841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.159267 31814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:41.159375 31814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "7abb46fb6bad4d91b1945ec2b7c3992a"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "8940d67b9de0d1110ff637537b803f83"
server_key_iv: "808db41976afe5376468db39f21bd5ff"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.159449 31814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:41.173713 31814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:41.174081 31814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:41.174242 31814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:41.174482 31814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:41.174820 31814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:41.174854 31814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.174907 31814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:41.174937 31814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.180668 31814 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:39199
I20260810 04:46:41.180724 32039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:39199 every 8 connection(s)
I20260810 04:46:41.181088 31814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:46:41.186148 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.181866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:44181 (local address 127.14.237.62:34607)
0810 04:46:41.181968 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.181970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.182192 (+ 222us) server_negotiation.cc:407] Connection header received
0810 04:46:41.182672 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.182673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.182684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.182740 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.183374 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.183838 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.184797 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.184977 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.185027 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:41.185946 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:41.185947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:41.185948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:41.185952 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:41.185963 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:41.185967 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:41.186008 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:41.186070 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 04:46:41.186091 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 04:46:41.186338 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.181823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34607 (local address 127.14.237.2:44181)
0810 04:46:41.182055 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:41.182087 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:41.182586 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:41.182793 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:41.182809 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:41.182816 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:41.183307 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:41.183311 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.183920 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.183921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:41.184719 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:41.184721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.185053 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.185054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:41.185198 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:41.185823 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:41.185824 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:41.185852 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:41.185874 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:41.186009 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:41.186011 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:41.186013 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:46:41.186059 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 04:46:41.186140 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":123,"threads_started":1}
I20260810 04:46:41.186923 32040 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34607
I20260810 04:46:41.187002 32040 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:41.187012 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 31814
I20260810 04:46:41.187103 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:41.187211 32040 heartbeater.cc:511] Master 127.14.237.62:34607 requested a full tablet report, sending...
I20260810 04:46:41.187328 15284 external_mini_cluster.cc:1428] Setting key a36afc51b7cafb3b25dc1d7951aa15a9
I20260810 04:46:41.187624 31488 ts_manager.cc:194] Registered new tserver with Master: 7abb46fb6bad4d91b1945ec2b7c3992a (127.14.237.2:39199)
I20260810 04:46:41.188220 31488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.2:44181
I20260810 04:46:41.188683 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34607
--builtin_ntp_servers=127.14.237.20:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 04:46:41.271500 32049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:41.271697 32049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:41.271718 32049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:41.272882 32049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:41.272922 32049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:41.272948 32049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:41.274362 32049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:41.274595 32049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:41.274894 32049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:41.279793 32059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.279771 32062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.279789 32060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.280040 32049 server_base.cc:1034] running on GCE node
I20260810 04:46:41.280262 32049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:41.280514 32049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:41.281672 32049 hybrid_clock.cc:648] HybridClock initialized: now 1786337201281660 us; error 31 us; skew 500 ppm
I20260810 04:46:41.282912 32049 webserver.cc:460] Webserver started at http://127.14.237.3:40171/ using document root <none> and password file <none>
I20260810 04:46:41.283205 32049 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:41.283277 32049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:41.283435 32049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:41.284479 32049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "7c6653a710004d9e99bbf0b044c86dec"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "4a573fafde3f8bb456e2abc13ccf0c2c"
server_key_iv: "a80a1446d9cb8d63eee9d4c1b4d75799"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.284842 32049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "7c6653a710004d9e99bbf0b044c86dec"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "4a573fafde3f8bb456e2abc13ccf0c2c"
server_key_iv: "a80a1446d9cb8d63eee9d4c1b4d75799"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.286259 32049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:41.287110 32071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.287288 32049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:41.287385 32049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "7c6653a710004d9e99bbf0b044c86dec"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "4a573fafde3f8bb456e2abc13ccf0c2c"
server_key_iv: "a80a1446d9cb8d63eee9d4c1b4d75799"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.287467 32049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:41.303520 32049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:41.303872 32049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:41.304062 32049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:41.304327 32049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:41.304661 32049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:41.304716 32049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.304750 32049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:41.304805 32049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.310521 32049 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:43939
I20260810 04:46:41.310564 32264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:43939 every 8 connection(s)
I20260810 04:46:41.310925 32049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:41.314316 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 32049
I20260810 04:46:41.314399 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:41.314595 15284 external_mini_cluster.cc:1428] Setting key 607d1585f415a19e7cc881eb16e52606
I20260810 04:46:41.316684 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.311762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:34279 (local address 127.14.237.62:34607)
0810 04:46:41.311849 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.311852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.312061 (+ 209us) server_negotiation.cc:407] Connection header received
0810 04:46:41.312574 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.312575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.312585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.312639 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.313351 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.313837 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.315209 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.315403 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.315451 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:41.316479 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:41.316480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:41.316480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:41.316484 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:41.316496 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:41.316500 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:41.316540 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:41.316606 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 04:46:41.316627 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 04:46:41.316972 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.311716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:34607 (local address 127.14.237.3:34279)
0810 04:46:41.311944 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:41.311977 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:41.312492 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:41.312694 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:41.312697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:41.312713 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:41.313273 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:41.313279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.313950 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.313951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:41.315113 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:41.315117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:41.315487 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:41.315488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:41.315685 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:41.316348 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:41.316349 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:41.316379 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:41.316403 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:41.316543 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:41.316546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:41.316549 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:41.316607 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:46:41.316684 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":115,"threads_started":1}
I20260810 04:46:41.317631 32265 heartbeater.cc:347] Connected to a master server at 127.14.237.62:34607
I20260810 04:46:41.317741 32265 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:41.317967 32265 heartbeater.cc:511] Master 127.14.237.62:34607 requested a full tablet report, sending...
I20260810 04:46:41.318375 31488 ts_manager.cc:194] Registered new tserver with Master: 7c6653a710004d9e99bbf0b044c86dec (127.14.237.3:43939)
I20260810 04:46:41.318725 31488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.3:34279
I20260810 04:46:41.326542 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:46:41.328418 31577 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:41.327848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.14.237.62:34607)
0810 04:46:41.327952 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.327954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.328043 (+ 89us) server_negotiation.cc:407] Connection header received
0810 04:46:41.328131 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.328132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.328145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.328201 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.328350 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57222: BlockingRecv error: recv got EOF from 127.0.0.1:57222 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 04:46:41.328490 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 31581
I20260810 04:46:41.352272 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 31814
I20260810 04:46:41.365368 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 32049
I20260810 04:46:41.374814 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 31443
2026-08-10T04:46:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (648 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T04:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:41Z Disabled control of system clock
I20260810 04:46:41.406031 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:40609
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:40609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:46:41.488109 32278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:41.488296 32278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:41.488315 32278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:41.489516 32278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:41.489557 32278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:41.489568 32278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:41.489614 32278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:41.489646 32278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:41.491004 32278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:40609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:40609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:41.491258 32278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:41.491545 32278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:41.496513 32292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.496505 32284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.496546 32278 server_base.cc:1034] running on GCE node
W20260810 04:46:41.496519 32287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.496992 32278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:41.497329 32278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:41.498497 32278 hybrid_clock.cc:648] HybridClock initialized: now 1786337201498482 us; error 28 us; skew 500 ppm
I20260810 04:46:41.499752 32278 webserver.cc:460] Webserver started at http://127.14.237.62:41639/ using document root <none> and password file <none>
I20260810 04:46:41.500031 32278 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:41.500098 32278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:41.500237 32278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:41.501317 32278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "c1dc011d96dc459da66c8d5cd2e86a86"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "cc50ec80cdde2d5ee25fe5fce3e79ef6"
server_key_iv: "85b290d7b0935ba421e68c9c6811348d"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.501714 32278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "c1dc011d96dc459da66c8d5cd2e86a86"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "cc50ec80cdde2d5ee25fe5fce3e79ef6"
server_key_iv: "85b290d7b0935ba421e68c9c6811348d"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.502982 32278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:41.503710 32304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.504101 32278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:41.504228 32278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "c1dc011d96dc459da66c8d5cd2e86a86"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "cc50ec80cdde2d5ee25fe5fce3e79ef6"
server_key_iv: "85b290d7b0935ba421e68c9c6811348d"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.504313 32278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:41.507823 32278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:41.508108 32278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:41.508265 32278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:41.512601 32278 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:40609
I20260810 04:46:41.512636 32392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:40609 every 8 connection(s)
I20260810 04:46:41.512966 32278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:41.513660 32393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:41.516245 32393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86: Bootstrap starting.
I20260810 04:46:41.516753 32393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:41.516994 32393 log.cc:824] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:41.517683 32393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86: No bootstrap required, opened a new log
I20260810 04:46:41.519201 32393 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dc011d96dc459da66c8d5cd2e86a86" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40609 } }
I20260810 04:46:41.519326 32393 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:41.519348 32393 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1dc011d96dc459da66c8d5cd2e86a86, State: Initialized, Role: FOLLOWER
I20260810 04:46:41.519480 32393 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dc011d96dc459da66c8d5cd2e86a86" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40609 } }
I20260810 04:46:41.519557 32393 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:41.519580 32393 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:41.519606 32393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:41.520087 32393 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dc011d96dc459da66c8d5cd2e86a86" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40609 } }
I20260810 04:46:41.520176 32393 leader_election.cc:302] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1dc011d96dc459da66c8d5cd2e86a86; no voters:
I20260810 04:46:41.520330 32393 leader_election.cc:288] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:41.520407 32402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:41.520429 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 32278
I20260810 04:46:41.520524 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:41.520599 32393 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:41.520684 32402 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [term 1 LEADER]: Becoming Leader. State: Replica: c1dc011d96dc459da66c8d5cd2e86a86, State: Running, Role: LEADER
I20260810 04:46:41.520778 15284 external_mini_cluster.cc:1428] Setting key e67ac6aae7f40774c875cfd6c9cdb4dc
I20260810 04:46:41.520803 32402 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dc011d96dc459da66c8d5cd2e86a86" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40609 } }
I20260810 04:46:41.521641 32405 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1dc011d96dc459da66c8d5cd2e86a86. Latest consensus state: current_term: 1 leader_uuid: "c1dc011d96dc459da66c8d5cd2e86a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dc011d96dc459da66c8d5cd2e86a86" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40609 } } }
I20260810 04:46:41.521741 32405 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:41.522086 32404 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1dc011d96dc459da66c8d5cd2e86a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dc011d96dc459da66c8d5cd2e86a86" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40609 } } }
I20260810 04:46:41.522168 32404 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86 [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:41.522459 32418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:41.523036 32418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:41.523494 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.521296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.14.237.62:40609)
0810 04:46:41.521564 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.521573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.521615 (+ 42us) server_negotiation.cc:407] Connection header received
0810 04:46:41.521862 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.521870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.521918 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.522422 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.522818 (+ 396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:41.522822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:41.522822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:41.522826 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:41.522919 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:41.522986 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:41.523055 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:41.523238 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 04:46:41.523312 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260810 04:46:41.524788 32418 catalog_manager.cc:1347] Generated new cluster ID: 8189e8018ab049929679c1b83e9bec14
I20260810 04:46:41.524837 32418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:46:41.542129 32418 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:41.542356 32418 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:41.544955 32418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1dc011d96dc459da66c8d5cd2e86a86: Generated new TSK 0
I20260810 04:46:41.545164 32418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:46:41.548415 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40609
--builtin_ntp_servers=127.14.237.20:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:46:41.632366 32435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:41.632576 32435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:41.632599 32435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:41.633872 32435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:41.633914 32435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:41.633940 32435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:41.635310 32435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:41.635584 32435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:41.635922 32435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:41.641716 32446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.641812 32448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.641707 32445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.642092 32435 server_base.cc:1034] running on GCE node
I20260810 04:46:41.642335 32435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:41.642575 32435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:41.643754 32435 hybrid_clock.cc:648] HybridClock initialized: now 1786337201643731 us; error 37 us; skew 500 ppm
I20260810 04:46:41.645037 32435 webserver.cc:460] Webserver started at http://127.14.237.1:46745/ using document root <none> and password file <none>
I20260810 04:46:41.645439 32435 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:41.645516 32435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:41.645671 32435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:41.646776 32435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "8cd66ba8e7cd479e8080f1ace892c1e0"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "37e3e237af2655b4fe497d5a6a596533"
server_key_iv: "3a0e190122c22fb7b5d71efd6c191b1d"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.647166 32435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "8cd66ba8e7cd479e8080f1ace892c1e0"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "37e3e237af2655b4fe497d5a6a596533"
server_key_iv: "3a0e190122c22fb7b5d71efd6c191b1d"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.648540 32435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:41.649366 32457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.649556 32435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:46:41.649677 32435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "8cd66ba8e7cd479e8080f1ace892c1e0"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "37e3e237af2655b4fe497d5a6a596533"
server_key_iv: "3a0e190122c22fb7b5d71efd6c191b1d"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.649755 32435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:41.653131 32435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:41.653434 32435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:41.653584 32435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:41.653810 32435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:41.654137 32435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:41.654194 32435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.654235 32435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:41.654261 32435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.660209 32435 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:38079
I20260810 04:46:41.660254 32634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:38079 every 8 connection(s)
I20260810 04:46:41.660614 32435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:46:41.663558 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.661406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:60167 (local address 127.14.237.62:40609)
0810 04:46:41.661491 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.661494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.661705 (+ 211us) server_negotiation.cc:407] Connection header received
0810 04:46:41.662183 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.662184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.662193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.662247 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.663326 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:41.663328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:41.663328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:41.663332 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:41.663347 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:41.663351 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:41.663399 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:41.663488 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 04:46:41.663503 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 04:46:41.663759 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.661386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40609 (local address 127.14.237.1:60167)
0810 04:46:41.661576 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:41.661609 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:41.662073 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:41.662300 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:41.662303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:41.662311 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:41.663177 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:41.663178 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:41.663205 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:41.663242 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:41.663417 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:41.663420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:41.663422 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:46:41.663475 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:46:41.663523 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260810 04:46:41.664180 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 32435
I20260810 04:46:41.664292 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:41.664407 32638 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40609
I20260810 04:46:41.664505 32638 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:41.664520 15284 external_mini_cluster.cc:1428] Setting key 1dc9c81d850c7f9ed463577040734f19
I20260810 04:46:41.664664 32638 heartbeater.cc:511] Master 127.14.237.62:40609 requested a full tablet report, sending...
I20260810 04:46:41.665268 32333 ts_manager.cc:194] Registered new tserver with Master: 8cd66ba8e7cd479e8080f1ace892c1e0 (127.14.237.1:38079)
I20260810 04:46:41.665571 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40609
--builtin_ntp_servers=127.14.237.20:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:46:41.748672 32645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:41.748864 32645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:41.748884 32645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:41.750094 32645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:41.750136 32645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:41.750161 32645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:41.751513 32645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:41.751753 32645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:41.752070 32645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:41.757081 32656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.757056 32653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.757119 32654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.757155 32645 server_base.cc:1034] running on GCE node
I20260810 04:46:41.757452 32645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:41.757628 32645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:41.758826 32645 hybrid_clock.cc:648] HybridClock initialized: now 1786337201758818 us; error 41 us; skew 500 ppm
I20260810 04:46:41.759999 32645 webserver.cc:460] Webserver started at http://127.14.237.2:37109/ using document root <none> and password file <none>
I20260810 04:46:41.760195 32645 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:41.760245 32645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:41.760351 32645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:41.761458 32645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "dee1c739a83244ea9b3f1858baa01a16"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "0388236a84a4becdb6da1ed480c2fc75"
server_key_iv: "5fb222ab89858a6b315e19d3b0d19096"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.761776 32645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "dee1c739a83244ea9b3f1858baa01a16"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "0388236a84a4becdb6da1ed480c2fc75"
server_key_iv: "5fb222ab89858a6b315e19d3b0d19096"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.763025 32645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:46:41.763685 32668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.763864 32645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:41.763972 32645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "dee1c739a83244ea9b3f1858baa01a16"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "0388236a84a4becdb6da1ed480c2fc75"
server_key_iv: "5fb222ab89858a6b315e19d3b0d19096"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.764047 32645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:41.767372 32645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:41.767654 32645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:41.767807 32645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:41.768025 32645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:41.768323 32645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:41.768378 32645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.768415 32645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:41.768443 32645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.774295 32645 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:37421
I20260810 04:46:41.774333 374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:37421 every 8 connection(s)
I20260810 04:46:41.774675 32645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:46:41.777864 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.775446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:39287 (local address 127.14.237.62:40609)
0810 04:46:41.775511 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.775513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.775724 (+ 211us) server_negotiation.cc:407] Connection header received
0810 04:46:41.776241 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.776243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.776250 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.776304 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.777565 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:41.777567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:41.777567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:41.777572 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:41.777590 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:41.777595 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:41.777680 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:41.777780 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 04:46:41.777796 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:46:41.778049 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.775435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40609 (local address 127.14.237.2:39287)
0810 04:46:41.775608 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:41.775641 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:41.776128 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:41.776387 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:41.776391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:41.776398 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:41.777412 (+ 1014us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:41.777413 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:41.777439 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:41.777465 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:41.777681 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:41.777687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:41.777690 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:41.777777 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:46:41.777838 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260810 04:46:41.778612 376 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40609
I20260810 04:46:41.778728 376 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:41.778890 376 heartbeater.cc:511] Master 127.14.237.62:40609 requested a full tablet report, sending...
I20260810 04:46:41.779317 32333 ts_manager.cc:194] Registered new tserver with Master: dee1c739a83244ea9b3f1858baa01a16 (127.14.237.2:37421)
I20260810 04:46:41.781170 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 32645
I20260810 04:46:41.781308 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:41.781473 15284 external_mini_cluster.cc:1428] Setting key 29a20940ae8e94e79cf034feaae8d65f
I20260810 04:46:41.782464 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40609
--builtin_ntp_servers=127.14.237.20:40089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:46:41.869490 382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:41.869691 382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:41.869711 382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:41.870890 382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:41.870932 382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:41.870957 382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:41.872318 382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:40089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:41.872558 382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:41.872848 382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:41.877842 392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:41.877827 394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.877859 382 server_base.cc:1034] running on GCE node
W20260810 04:46:41.877835 389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:41.878199 382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:41.878446 382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:41.879616 382 hybrid_clock.cc:648] HybridClock initialized: now 1786337201879601 us; error 23 us; skew 500 ppm
I20260810 04:46:41.880888 382 webserver.cc:460] Webserver started at http://127.14.237.3:45223/ using document root <none> and password file <none>
I20260810 04:46:41.881145 382 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:41.881217 382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:41.881399 382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:41.882411 382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "412c946fac5e4d95b95ea91e816eaea9"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "3d24f15c37ef04e1372b63dc43567001"
server_key_iv: "b7973afafda329c4a340619c0903ea63"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.882817 382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "412c946fac5e4d95b95ea91e816eaea9"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "3d24f15c37ef04e1372b63dc43567001"
server_key_iv: "b7973afafda329c4a340619c0903ea63"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.884171 382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:41.884879 401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.885056 382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:46:41.885246 382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "412c946fac5e4d95b95ea91e816eaea9"
format_stamp: "Formatted at 2026-08-10 04:46:41 on dist-test-slave-8hhm"
server_key: "3d24f15c37ef04e1372b63dc43567001"
server_key_iv: "b7973afafda329c4a340619c0903ea63"
server_key_version: "encryptionkey@0"
I20260810 04:46:41.885339 382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:41.888688 382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:41.888938 382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:41.889087 382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:41.889322 382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:41.889640 382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:41.889698 382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.889739 382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:41.889765 382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:41.895639 382 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:37165
I20260810 04:46:41.895676 588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:37165 every 8 connection(s)
I20260810 04:46:41.896021 382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:46:41.897774 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 382
I20260810 04:46:41.897874 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:41.898077 15284 external_mini_cluster.cc:1428] Setting key 170edb761dc52ecb1d0149f6697c5a2b
I20260810 04:46:41.899358 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.896728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:60499 (local address 127.14.237.62:40609)
0810 04:46:41.896773 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.896775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.897004 (+ 229us) server_negotiation.cc:407] Connection header received
0810 04:46:41.897561 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.897562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.897569 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.897637 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.899044 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:41.899045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:41.899045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:46:41.899050 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:41.899095 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:46:41.899106 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:41.899174 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:41.899260 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:46:41.899288 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:46:41.899566 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:41.896701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40609 (local address 127.14.237.3:60499)
0810 04:46:41.896868 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:41.896923 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:41.897434 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:41.897709 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:41.897714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:41.897734 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:41.898872 (+ 1138us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:46:41.898873 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:41.898910 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:46:41.898943 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:41.899174 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:41.899182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:41.899185 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:41.899282 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 04:46:41.899340 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260810 04:46:41.899991 590 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40609
I20260810 04:46:41.900084 590 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:41.900221 590 heartbeater.cc:511] Master 127.14.237.62:40609 requested a full tablet report, sending...
I20260810 04:46:41.900550 32333 ts_manager.cc:194] Registered new tserver with Master: 412c946fac5e4d95b95ea91e816eaea9 (127.14.237.3:37165)
I20260810 04:46:41.909929 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:46:41.911875 32414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:46:41.911293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.14.237.62:40609)
0810 04:46:41.911367 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:41.911370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:41.911481 (+ 111us) server_negotiation.cc:407] Connection header received
0810 04:46:41.911559 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:41.911560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:41.911565 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:41.911642 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:41.911812 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42086: BlockingRecv error: recv got EOF from 127.0.0.1:42086 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:46:41.911934 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 32435
I20260810 04:46:41.927950 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 32645
I20260810 04:46:41.942622 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 382
I20260810 04:46:41.956283 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 32278
2026-08-10T04:46:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (586 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:42 dist-test-slave-8hhm krb5kdc[609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:42 dist-test-slave-8hhm krb5kdc[609](info): set up 2 sockets
Aug 10 04:46:42 dist-test-slave-8hhm krb5kdc[609](info): commencing operation
krb5kdc: starting...
W20260810 04:46:43.214110 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:46:43.332067 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:39011
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:39011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:43.415021 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:43.415239 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:43.415263 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:43.416344 632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:46:43.416383 632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:43.416399 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:43.416414 632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:46:43.416426 632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:46:43.417907 632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:39011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:39011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:43.418114 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:43.418305 632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:43.424479 647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:43.424544 645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:43.424494 642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:43.424844 632 server_base.cc:1034] running on GCE node
I20260810 04:46:43.425117 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:43.425472 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:43.426635 632 hybrid_clock.cc:648] HybridClock initialized: now 1786337203426614 us; error 41 us; skew 500 ppm
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:43.429122 632 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:46:43.429566 632 webserver.cc:460] Webserver started at http://127.14.237.62:36763/ using document root <none> and password file <none>
I20260810 04:46:43.429816 632 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:43.429873 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:43.430007 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:43.431083 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "1a9b85f2644c4a0d9c993b0534b7412f"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "13f5d0705e45268ed5f10286972f8d04"
server_key_iv: "ca81cb7f5127e83560d34ceeb3da82c8"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.431440 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "1a9b85f2644c4a0d9c993b0534b7412f"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "13f5d0705e45268ed5f10286972f8d04"
server_key_iv: "ca81cb7f5127e83560d34ceeb3da82c8"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.432822 632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:46:43.433658 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.433882 632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:46:43.433977 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "1a9b85f2644c4a0d9c993b0534b7412f"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "13f5d0705e45268ed5f10286972f8d04"
server_key_iv: "ca81cb7f5127e83560d34ceeb3da82c8"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.434051 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:43.440371 632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:43.442359 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:43.442503 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:43.446560 632 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:39011
I20260810 04:46:43.446630 733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:39011 every 8 connection(s)
I20260810 04:46:43.446969 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:46:43.447953 735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:43.450222 735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f: Bootstrap starting.
I20260810 04:46:43.450760 735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:43.451009 735 log.cc:824] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:43.451678 735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f: No bootstrap required, opened a new log
I20260810 04:46:43.453302 735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b85f2644c4a0d9c993b0534b7412f" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39011 } }
I20260810 04:46:43.453444 735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:43.453466 735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9b85f2644c4a0d9c993b0534b7412f, State: Initialized, Role: FOLLOWER
I20260810 04:46:43.453562 735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b85f2644c4a0d9c993b0534b7412f" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39011 } }
I20260810 04:46:43.453653 735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:46:43.453688 735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:46:43.453760 735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:43.454542 735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b85f2644c4a0d9c993b0534b7412f" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39011 } }
I20260810 04:46:43.454684 735 leader_election.cc:302] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a9b85f2644c4a0d9c993b0534b7412f; no voters:
I20260810 04:46:43.454852 735 leader_election.cc:288] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:46:43.454881 741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:43.455044 741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [term 1 LEADER]: Becoming Leader. State: Replica: 1a9b85f2644c4a0d9c993b0534b7412f, State: Running, Role: LEADER
I20260810 04:46:43.455127 735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:46:43.455134 741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b85f2644c4a0d9c993b0534b7412f" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39011 } }
I20260810 04:46:43.455571 744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9b85f2644c4a0d9c993b0534b7412f. Latest consensus state: current_term: 1 leader_uuid: "1a9b85f2644c4a0d9c993b0534b7412f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b85f2644c4a0d9c993b0534b7412f" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39011 } } }
I20260810 04:46:43.455603 742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a9b85f2644c4a0d9c993b0534b7412f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9b85f2644c4a0d9c993b0534b7412f" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 39011 } } }
I20260810 04:46:43.455678 744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:43.455681 742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f [sys.catalog]: This master's current role is: LEADER
I20260810 04:46:43.456332 763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:46:43.456763 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 632
I20260810 04:46:43.456883 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:46:43.456892 763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:46:43.457096 15284 external_mini_cluster.cc:1428] Setting key 39dffa5a746f0ca4ffdb28acbd05a72e
I20260810 04:46:43.458525 763 catalog_manager.cc:1347] Generated new cluster ID: 50f279f69e5e4fccbc9c3d0242c7f11f
I20260810 04:46:43.458580 763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:43.469247 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.457592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.14.237.62:39011)
0810 04:46:43.457776 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:43.457785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:43.458191 (+ 406us) server_negotiation.cc:407] Connection header received
0810 04:46:43.458310 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:43.458317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:43.458383 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:43.458551 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:43.459350 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.460735 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.461857 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.462481 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.462587 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.465469 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:43.465472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:43.465472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:43.465483 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:43.467628 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.468021 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.468024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.468024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.468144 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.468347 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.468347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.468348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.468639 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:43.468788 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:43.468857 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 04:46:43.468988 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260810 04:46:43.482831 763 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:46:43.483414 763 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:46:43.485848 763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f: Generated new TSK 0
I20260810 04:46:43.486034 763 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:46:43.513178 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:39011
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:43.594596 796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:43.594774 796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:43.594795 796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:43.595917 796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:43.595959 796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:43.595984 796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:46:43.597501 796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:39011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:43.597731 796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:43.598021 796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:43.602934 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:43.602939 805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:43.603039 804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:43.603147 796 server_base.cc:1034] running on GCE node
I20260810 04:46:43.603405 796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:43.603595 796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:43.604719 796 hybrid_clock.cc:648] HybridClock initialized: now 1786337203604691 us; error 39 us; skew 500 ppm
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:43.607352 796 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:46:43.607801 796 webserver.cc:460] Webserver started at http://127.14.237.1:41393/ using document root <none> and password file <none>
I20260810 04:46:43.608052 796 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:43.608096 796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:43.608237 796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:43.609321 796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "19b6e50b050b49d88ce8af7b760e75d0"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "293ef11da6f43c36103c5057fdc543dd"
server_key_iv: "f454a61560e70ebb7e560a71f1e05b61"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.609675 796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "19b6e50b050b49d88ce8af7b760e75d0"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "293ef11da6f43c36103c5057fdc543dd"
server_key_iv: "f454a61560e70ebb7e560a71f1e05b61"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.610935 796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:43.611632 818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.611828 796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:43.611894 796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "19b6e50b050b49d88ce8af7b760e75d0"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "293ef11da6f43c36103c5057fdc543dd"
server_key_iv: "f454a61560e70ebb7e560a71f1e05b61"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.611994 796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:43.621310 796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:43.623037 796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:43.623174 796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:43.623396 796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:43.623678 796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:43.623713 796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.623755 796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:43.623780 796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.629654 796 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:34853
I20260810 04:46:43.629707 989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:34853 every 8 connection(s)
I20260810 04:46:43.629983 796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:43.637744 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 796
I20260810 04:46:43.637850 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:46:43.638096 15284 external_mini_cluster.cc:1428] Setting key 0314db378cde161c3a167a7dd7ef69f7
I20260810 04:46:43.639089 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.630672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:35397 (local address 127.14.237.62:39011)
0810 04:46:43.630714 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:43.630716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:43.630920 (+ 204us) server_negotiation.cc:407] Connection header received
0810 04:46:43.631376 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:43.631377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:43.631390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:43.631445 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:43.632329 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.632816 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.633872 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.634080 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.634134 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.636255 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:43.636257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:43.636257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:43.636265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:43.637619 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.638167 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.638169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.638169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.638237 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.638470 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.638471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.638471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.638602 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:43.638679 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:43.638950 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 04:46:43.639026 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:46:43.639331 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.630638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:39011 (local address 127.14.237.1:35397)
0810 04:46:43.630797 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:43.630836 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:43.631297 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:43.631494 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:43.631497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:43.632058 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:43.632246 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.632250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.632898 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.632899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.633795 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.633799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.634197 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.634198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.634303 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.634756 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:43.634764 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:43.636163 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:43.637704 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:43.637720 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:43.637722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:43.638054 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:43.638305 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:43.638306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:43.638306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:43.638417 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:43.638675 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:43.638680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:43.638791 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 04:46:43.638922 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 04:46:43.639093 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260810 04:46:43.639935 990 heartbeater.cc:347] Connected to a master server at 127.14.237.62:39011
I20260810 04:46:43.640028 990 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:43.640223 990 heartbeater.cc:511] Master 127.14.237.62:39011 requested a full tablet report, sending...
I20260810 04:46:43.640745 683 ts_manager.cc:194] Registered new tserver with Master: 19b6e50b050b49d88ce8af7b760e75d0 (127.14.237.1:34853)
I20260810 04:46:43.641430 683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:35397
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:46:43.663789 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:39011
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:43.745018 1002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:43.745265 1002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:43.745306 1002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:43.746438 1002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:43.746475 1002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:43.746529 1002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:46:43.747967 1002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:39011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:43.748210 1002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:43.748436 1002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:43.753528 1013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:43.753539 1009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:43.753539 1008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:43.753789 1002 server_base.cc:1034] running on GCE node
I20260810 04:46:43.754031 1002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:43.754271 1002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:43.755424 1002 hybrid_clock.cc:648] HybridClock initialized: now 1786337203755413 us; error 34 us; skew 500 ppm
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:43.758077 1002 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:46:43.758554 1002 webserver.cc:460] Webserver started at http://127.14.237.2:40445/ using document root <none> and password file <none>
I20260810 04:46:43.758805 1002 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:43.758872 1002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:43.759016 1002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:43.760080 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "950a9fcafec340f190dbbd391b0a5abc"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "c61c7b1d38c1711591ee6b551321e123"
server_key_iv: "6970adfe3201ba02642c62b55c1e2b82"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.760448 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "950a9fcafec340f190dbbd391b0a5abc"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "c61c7b1d38c1711591ee6b551321e123"
server_key_iv: "6970adfe3201ba02642c62b55c1e2b82"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.761859 1002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:43.762565 1027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.762743 1002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:46:43.762848 1002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "950a9fcafec340f190dbbd391b0a5abc"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "c61c7b1d38c1711591ee6b551321e123"
server_key_iv: "6970adfe3201ba02642c62b55c1e2b82"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.762920 1002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:43.772835 1002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:43.774792 1002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:43.774955 1002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:43.775171 1002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:43.775496 1002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:43.775548 1002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.775583 1002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:43.775609 1002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.781289 1002 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:41453
I20260810 04:46:43.781352 1201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:41453 every 8 connection(s)
I20260810 04:46:43.781669 1002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:43.788847 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 1002
I20260810 04:46:43.788934 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:46:43.789258 15284 external_mini_cluster.cc:1428] Setting key ec36513712eb5b3fbbc4417f390bcb09
I20260810 04:46:43.791357 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.782448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:59777 (local address 127.14.237.62:39011)
0810 04:46:43.782509 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:43.782511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:43.782687 (+ 176us) server_negotiation.cc:407] Connection header received
0810 04:46:43.783184 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:43.783185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:43.783194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:43.783259 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:43.784182 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.784628 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.785760 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.785962 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.786019 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.788225 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:43.788226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:43.788226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:43.788233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:43.789743 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.790249 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.790250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.790251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.790315 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.790563 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.790564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.790565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.790711 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:43.790803 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:43.791119 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 04:46:43.791210 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:46:43.791443 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.782384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:39011 (local address 127.14.237.2:59777)
0810 04:46:43.782557 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:43.782597 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:43.783097 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:43.783312 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:43.783315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:43.783891 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:43.784085 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.784091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.784740 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.784741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.785675 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.785678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.786038 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.786039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.786171 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.786620 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:43.786628 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:43.788128 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:43.789816 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:43.789823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:43.789825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:43.790156 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:43.790390 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:43.790391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:43.790391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:43.790495 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:43.790834 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:43.790841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:43.790974 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:46:43.791078 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 04:46:43.791238 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260810 04:46:43.791975 1204 heartbeater.cc:347] Connected to a master server at 127.14.237.62:39011
I20260810 04:46:43.792083 1204 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:43.792280 1204 heartbeater.cc:511] Master 127.14.237.62:39011 requested a full tablet report, sending...
I20260810 04:46:43.792761 683 ts_manager.cc:194] Registered new tserver with Master: 950a9fcafec340f190dbbd391b0a5abc (127.14.237.2:41453)
I20260810 04:46:43.793282 683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:59777
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:46:43.815407 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:39011
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:46:43.898007 1217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:46:43.898193 1217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:46:43.898216 1217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:46:43.899358 1217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:46:43.899397 1217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:46:43.899423 1217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:46:43.900906 1217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:39011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:46:43.901083 1217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:46:43.901292 1217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:46:43.906014 1229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:43.906038 1225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:46:43.906021 1226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:46:43.906419 1217 server_base.cc:1034] running on GCE node
I20260810 04:46:43.906642 1217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:46:43.906901 1217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:46:43.908052 1217 hybrid_clock.cc:648] HybridClock initialized: now 1786337203908025 us; error 42 us; skew 500 ppm
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:46:43.910663 1217 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:46:43.911162 1217 webserver.cc:460] Webserver started at http://127.14.237.3:40789/ using document root <none> and password file <none>
I20260810 04:46:43.911417 1217 fs_manager.cc:359] Metadata directory not provided
I20260810 04:46:43.911482 1217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:46:43.911633 1217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:46:43.912637 1217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "eb1c66a9efa44d8f8629efe330c8a011"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "4976512ce2284f491b9bce21a2617f5c"
server_key_iv: "4969070dc24e5bd98955e6335dcfd3a2"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.912997 1217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "eb1c66a9efa44d8f8629efe330c8a011"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "4976512ce2284f491b9bce21a2617f5c"
server_key_iv: "4969070dc24e5bd98955e6335dcfd3a2"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.914358 1217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:46:43.915202 1241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.915380 1217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:46:43.915481 1217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "eb1c66a9efa44d8f8629efe330c8a011"
format_stamp: "Formatted at 2026-08-10 04:46:43 on dist-test-slave-8hhm"
server_key: "4976512ce2284f491b9bce21a2617f5c"
server_key_iv: "4969070dc24e5bd98955e6335dcfd3a2"
server_key_version: "encryptionkey@0"
I20260810 04:46:43.915570 1217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:46:43.933959 1217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:46:43.935843 1217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:46:43.936015 1217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:46:43.936280 1217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:46:43.936632 1217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:46:43.936666 1217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.936689 1217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:46:43.936702 1217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:46:43.942950 1217 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:38137
I20260810 04:46:43.943126 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:38137 every 8 connection(s)
I20260810 04:46:43.943374 1217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:46:43 dist-test-slave-8hhm krb5kdc[609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337203, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:46:43.950775 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 1217
I20260810 04:46:43.950904 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:46:43.951156 15284 external_mini_cluster.cc:1428] Setting key 635c7b06c802656331b1e40b884b5576
I20260810 04:46:43.953356 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.944147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:46489 (local address 127.14.237.62:39011)
0810 04:46:43.944218 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:43.944228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:43.944414 (+ 186us) server_negotiation.cc:407] Connection header received
0810 04:46:43.944883 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:43.944884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:43.944894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:43.944948 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:43.945929 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.946417 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.947366 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.947535 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.947586 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.950075 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:43.950076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:43.950077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:43.950084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:43.951990 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.952459 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.952460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.952460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.952517 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.952756 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.952757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.952757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.952870 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:43.952921 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:43.953146 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 04:46:43.953208 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:46:43.953569 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.944134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:39011 (local address 127.14.237.3:46489)
0810 04:46:43.944286 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:43.944322 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:43.944801 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:43.945003 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:43.945006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:43.945635 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:46:43.945853 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.945859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.946498 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.946498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.947292 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.947294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.947616 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.947617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.947716 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.948283 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:46:43.948292 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:46:43.949798 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:46:43.952077 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:43.952084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:43.952086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:43.952388 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:43.952580 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:46:43.952580 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:46:43.952581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:46:43.952687 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:46:43.952927 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:46:43.952931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:46:43.953044 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 04:46:43.953122 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:46:43.953307 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260810 04:46:43.954021 1423 heartbeater.cc:347] Connected to a master server at 127.14.237.62:39011
I20260810 04:46:43.954141 1423 heartbeater.cc:464] Registering TS with master...
I20260810 04:46:43.954327 1423 heartbeater.cc:511] Master 127.14.237.62:39011 requested a full tablet report, sending...
I20260810 04:46:43.954711 683 ts_manager.cc:194] Registered new tserver with Master: eb1c66a9efa44d8f8629efe330c8a011 (127.14.237.3:38137)
I20260810 04:46:43.955084 683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:46489
I20260810 04:46:43.963145 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:46:43.970777 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.964568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.14.237.62:39011)
0810 04:46:43.964672 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:43.964673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:43.964703 (+ 30us) server_negotiation.cc:407] Connection header received
0810 04:46:43.964772 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:43.964773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:43.964782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:43.964831 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:46:43.965511 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.966057 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.966822 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.967034 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.967109 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.968292 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:46:43.968293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:46:43.968294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:46:43.968300 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:46:43.969812 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.970129 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.970129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.970130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.970182 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:46:43.970342 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:46:43.970343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:46:43.970343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:46:43.970470 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:46:43.970540 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:43.970621 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 04:46:43.970678 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 04:46:43.972343 683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:46:43.974045 683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:46:43.980893 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.976865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.14.237.2:41453)
0810 04:46:43.977068 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:43.977073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:43.977083 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:46:43.977127 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:43.977129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:43.977144 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:43.977424 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:43.977491 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.978510 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.979788 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.980472 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.980571 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.980617 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:43.980771 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:46:43.980812 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260810 04:46:43.980893 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.976660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:41453 (local address 127.0.0.1:48374)
0810 04:46:43.976851 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:43.976860 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:43.976900 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:43.977298 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:43.977299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:43.977302 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:43.977346 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.977347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.978621 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.978622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.979711 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.979712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.980581 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.980582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.980656 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.980670 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:46:43.980741 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:46:43.980793 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":51,"threads_started":1}
I20260810 04:46:43.980995 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.976335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:34853 (local address 127.0.0.1:38744)
0810 04:46:43.976659 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:43.976705 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:43.976782 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:43.977398 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:43.977399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:43.977401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:43.977456 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.977458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.978326 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.978327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.979609 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.979612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.980639 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.980641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.980700 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.980703 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:46:43.980789 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 04:46:43.980831 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":32,"thread_start_us":70,"threads_started":1}
I20260810 04:46:43.981088 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.976457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.14.237.1:34853)
0810 04:46:43.976667 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:43.976672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:43.977034 (+ 362us) server_negotiation.cc:407] Connection header received
0810 04:46:43.977198 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:43.977200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:43.977238 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:43.977307 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:43.977543 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.978226 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.979691 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.980464 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.980540 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.980604 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:43.980935 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 04:46:43.980990 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260810 04:46:43.981846 891 tablet_service.cc:1467] Processing CreateTablet for tablet 83001600c899496892772b466bd335c9 (DEFAULT_TABLE table=test-table [id=f835fa93e867448bb5356bcf5444ba07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:43.981977 1110 tablet_service.cc:1467] Processing CreateTablet for tablet 83001600c899496892772b466bd335c9 (DEFAULT_TABLE table=test-table [id=f835fa93e867448bb5356bcf5444ba07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:43.982132 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.976505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:38137 (local address 127.0.0.1:51002)
0810 04:46:43.976718 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:43.976740 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:43.976794 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:43.977141 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:43.977144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:43.977147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:43.977244 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.977246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.979787 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.979788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.980424 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:43.980425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.981896 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.981898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:43.981962 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.981966 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:46:43.982032 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:46:43.982071 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":64,"threads_started":1}
I20260810 04:46:43.982187 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:43.976632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.14.237.3:38137)
0810 04:46:43.976834 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:43.976839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:43.976851 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:46:43.976891 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:43.976893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:43.976910 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:43.977002 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:43.977337 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.978502 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.980529 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:43.981806 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:43.981886 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:43.981941 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:43.982063 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 04:46:43.982114 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260810 04:46:43.982277 891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83001600c899496892772b466bd335c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:43.982344 1110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83001600c899496892772b466bd335c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:43.984929 1323 tablet_service.cc:1467] Processing CreateTablet for tablet 83001600c899496892772b466bd335c9 (DEFAULT_TABLE table=test-table [id=f835fa93e867448bb5356bcf5444ba07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:46:43.985184 1452 tablet_bootstrap.cc:492] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Bootstrap starting.
I20260810 04:46:43.985296 1323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83001600c899496892772b466bd335c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:46:43.985376 1454 tablet_bootstrap.cc:492] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0: Bootstrap starting.
I20260810 04:46:43.985939 1454 tablet_bootstrap.cc:654] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:43.985934 1452 tablet_bootstrap.cc:654] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:43.986500 1454 log.cc:824] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:43.986559 1452 log.cc:824] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:43.987550 1454 tablet_bootstrap.cc:492] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0: No bootstrap required, opened a new log
I20260810 04:46:43.987586 1452 tablet_bootstrap.cc:492] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: No bootstrap required, opened a new log
I20260810 04:46:43.987680 1454 ts_tablet_manager.cc:1397] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:46:43.987797 1452 ts_tablet_manager.cc:1397] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 04:46:43.990026 1452 raft_consensus.cc:348] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:43.990211 1452 raft_consensus.cc:374] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:43.990258 1452 raft_consensus.cc:729] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950a9fcafec340f190dbbd391b0a5abc, State: Initialized, Role: FOLLOWER
I20260810 04:46:43.990401 1452 consensus_queue.cc:260] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:43.990358 1454 raft_consensus.cc:348] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:43.990536 1454 raft_consensus.cc:374] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:43.990572 1454 raft_consensus.cc:729] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b6e50b050b49d88ce8af7b760e75d0, State: Initialized, Role: FOLLOWER
I20260810 04:46:43.990706 1452 ts_tablet_manager.cc:1428] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:46:43.990742 1204 heartbeater.cc:503] Master 127.14.237.62:39011 was elected leader, sending a full tablet report...
I20260810 04:46:43.990782 1454 consensus_queue.cc:260] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:43.990888 1457 tablet_bootstrap.cc:492] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: Bootstrap starting.
I20260810 04:46:43.991066 1454 ts_tablet_manager.cc:1428] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:46:43.991103 990 heartbeater.cc:503] Master 127.14.237.62:39011 was elected leader, sending a full tablet report...
I20260810 04:46:43.992076 1457 tablet_bootstrap.cc:654] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: Neither blocks nor log segments found. Creating new log.
I20260810 04:46:43.992570 1457 log.cc:824] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: Log is configured to *not* fsync() on all Append() calls
I20260810 04:46:43.993314 1457 tablet_bootstrap.cc:492] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: No bootstrap required, opened a new log
I20260810 04:46:43.993409 1457 ts_tablet_manager.cc:1397] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:46:43.994799 1457 raft_consensus.cc:348] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:43.994910 1457 raft_consensus.cc:374] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:46:43.994961 1457 raft_consensus.cc:729] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1c66a9efa44d8f8629efe330c8a011, State: Initialized, Role: FOLLOWER
I20260810 04:46:43.995067 1457 consensus_queue.cc:260] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:43.995359 1423 heartbeater.cc:503] Master 127.14.237.62:39011 was elected leader, sending a full tablet report...
I20260810 04:46:43.995365 1457 ts_tablet_manager.cc:1428] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260810 04:46:44.032333 1208 tablet.cc:2369] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:46:44.059762 1458 raft_consensus.cc:482] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:46:44.059892 1458 raft_consensus.cc:504] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:44.060259 1458 leader_election.cc:288] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19b6e50b050b49d88ce8af7b760e75d0 (127.14.237.1:34853), eb1c66a9efa44d8f8629efe330c8a011 (127.14.237.3:38137)
I20260810 04:46:44.064375 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:44.060433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:34853 (local address 127.14.237.2:55711)
0810 04:46:44.060505 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:44.060518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:44.060576 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:44.060821 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:44.060822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:44.060825 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:44.060898 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:44.060900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.061870 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.061871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:44.063279 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:44.063280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.064062 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.064062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:44.064152 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:44.064154 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:46:44.064212 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:46:44.064280 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":109}
I20260810 04:46:44.064425 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:44.060698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:44915 (local address 127.14.237.3:38137)
0810 04:46:44.060739 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:44.060741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:44.060846 (+ 105us) server_negotiation.cc:407] Connection header received
0810 04:46:44.060896 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:44.060897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:44.060908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:44.060981 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:44.061111 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.061794 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.063381 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.063965 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.064046 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:44.064055 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:44.064274 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 04:46:44.064325 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:46:44.064435 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:44.060581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:55711 (local address 127.14.237.1:34853)
0810 04:46:44.060666 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:44.060668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:44.060677 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:44.060706 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:44.060707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:44.060732 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:44.060778 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:46:44.060987 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.061764 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.063396 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.063944 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.064046 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:44.064055 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:44.064274 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 04:46:44.064325 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 04:46:44.064383 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:44.060592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:38137 (local address 127.14.237.2:44915)
0810 04:46:44.060788 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:44.060798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:46:44.060846 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:46:44.060992 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:46:44.060993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:46:44.060994 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:46:44.061044 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:44.061045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.061870 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.061871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:44.063271 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:46:44.063275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.064062 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.064062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:46:44.064141 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:44.064150 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 04:46:44.064212 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 04:46:44.064301 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":111,"thread_start_us":124,"threads_started":1}
I20260810 04:46:44.064894 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83001600c899496892772b466bd335c9" candidate_uuid: "950a9fcafec340f190dbbd391b0a5abc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b6e50b050b49d88ce8af7b760e75d0" is_pre_election: true
I20260810 04:46:44.065019 1348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83001600c899496892772b466bd335c9" candidate_uuid: "950a9fcafec340f190dbbd391b0a5abc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1c66a9efa44d8f8629efe330c8a011" is_pre_election: true
I20260810 04:46:44.065095 923 raft_consensus.cc:2454] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 950a9fcafec340f190dbbd391b0a5abc in term 0.
I20260810 04:46:44.065169 1348 raft_consensus.cc:2454] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 950a9fcafec340f190dbbd391b0a5abc in term 0.
I20260810 04:46:44.065392 1029 leader_election.cc:302] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b6e50b050b49d88ce8af7b760e75d0, 950a9fcafec340f190dbbd391b0a5abc; no voters:
I20260810 04:46:44.065565 1458 raft_consensus.cc:2781] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:46:44.065651 1458 raft_consensus.cc:482] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:46:44.065670 1458 raft_consensus.cc:3037] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:44.066444 1458 raft_consensus.cc:504] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:44.066583 1458 leader_election.cc:288] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [CANDIDATE]: Term 1 election: Requested vote from peers 19b6e50b050b49d88ce8af7b760e75d0 (127.14.237.1:34853), eb1c66a9efa44d8f8629efe330c8a011 (127.14.237.3:38137)
I20260810 04:46:44.066802 923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83001600c899496892772b466bd335c9" candidate_uuid: "950a9fcafec340f190dbbd391b0a5abc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b6e50b050b49d88ce8af7b760e75d0"
I20260810 04:46:44.066831 1348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83001600c899496892772b466bd335c9" candidate_uuid: "950a9fcafec340f190dbbd391b0a5abc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb1c66a9efa44d8f8629efe330c8a011"
I20260810 04:46:44.066882 923 raft_consensus.cc:3037] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:44.066928 1348 raft_consensus.cc:3037] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:46:44.067615 923 raft_consensus.cc:2454] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 950a9fcafec340f190dbbd391b0a5abc in term 1.
I20260810 04:46:44.067639 1348 raft_consensus.cc:2454] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 950a9fcafec340f190dbbd391b0a5abc in term 1.
I20260810 04:46:44.067787 1029 leader_election.cc:302] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b6e50b050b49d88ce8af7b760e75d0, 950a9fcafec340f190dbbd391b0a5abc; no voters:
I20260810 04:46:44.067914 1458 raft_consensus.cc:2781] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:46:44.068099 1458 raft_consensus.cc:686] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 1 LEADER]: Becoming Leader. State: Replica: 950a9fcafec340f190dbbd391b0a5abc, State: Running, Role: LEADER
I20260810 04:46:44.068207 1458 consensus_queue.cc:237] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } }
I20260810 04:46:44.069101 682 catalog_manager.cc:5495] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc reported cstate change: term changed from 0 to 1, leader changed from <none> to 950a9fcafec340f190dbbd391b0a5abc (127.14.237.2). New cstate: current_term: 1 leader_uuid: "950a9fcafec340f190dbbd391b0a5abc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "950a9fcafec340f190dbbd391b0a5abc" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41453 } health_report { overall_health: HEALTHY } } }
I20260810 04:46:44.075095 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:44.072646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.14.237.2:41453)
0810 04:46:44.072712 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:46:44.072714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:46:44.072723 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:46:44.072753 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:46:44.072754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:46:44.072773 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:46:44.072865 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:46:44.072983 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.073516 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.074289 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:46:44.074485 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:46:44.074587 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:46:44.074761 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:46:44.074864 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:46:44.074961 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 04:46:44.074980 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 04:46:44.074999 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:46:44.076890 923 raft_consensus.cc:1261] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0 [term 1 FOLLOWER]: Refusing update from remote peer 950a9fcafec340f190dbbd391b0a5abc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:44.076946 1348 raft_consensus.cc:1261] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 1 FOLLOWER]: Refusing update from remote peer 950a9fcafec340f190dbbd391b0a5abc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:46:44.077229 1465 consensus_queue.cc:1035] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b6e50b050b49d88ce8af7b760e75d0" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 34853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:44.077353 1458 consensus_queue.cc:1035] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1c66a9efa44d8f8629efe330c8a011" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:46:44.079898 1471 mvcc.cc:205] Tried to move back new op lower bound from 7316837187896565760 to 7316837187864018944. Current Snapshot: MvccSnapshot[applied={T|T < 7316837187896565760}]
I20260810 04:46:44.080070 1474 mvcc.cc:205] Tried to move back new op lower bound from 7316837187896565760 to 7316837187864018944. Current Snapshot: MvccSnapshot[applied={T|T < 7316837187896565760}]
I20260810 04:46:44.080366 1473 mvcc.cc:205] Tried to move back new op lower bound from 7316837187896565760 to 7316837187864018944. Current Snapshot: MvccSnapshot[applied={T|T < 7316837187896565760 or (T in {7316837187896565760})}]
I20260810 04:46:44.082403 683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:44.082501 683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:46:44.083345 683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a9b85f2644c4a0d9c993b0534b7412f: Sending DeleteTablet for 3 replicas of tablet 83001600c899496892772b466bd335c9
I20260810 04:46:44.083690 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 83001600c899496892772b466bd335c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:44 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:38744
I20260810 04:46:44.083817 1323 tablet_service.cc:1514] Processing DeleteTablet for tablet 83001600c899496892772b466bd335c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:44 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:51002
I20260810 04:46:44.083925 1488 tablet_replica.cc:330] T 83001600c899496892772b466bd335c9 P 19b6e50b050b49d88ce8af7b760e75d0: stopping tablet replica
I20260810 04:46:44.083961 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 796
I20260810 04:46:44.084019 1110 tablet_service.cc:1514] Processing DeleteTablet for tablet 83001600c899496892772b466bd335c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:46:44 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:48374
I20260810 04:46:44.084326 1489 tablet_replica.cc:330] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: stopping tablet replica
I20260810 04:46:44.084491 1490 tablet_replica.cc:330] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: stopping tablet replica
I20260810 04:46:44.084621 1490 raft_consensus.cc:2229] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:46:44.084750 1489 raft_consensus.cc:2229] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:46:44.084883 1489 raft_consensus.cc:2258] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:44.085003 1490 raft_consensus.cc:2258] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:46:44.087517 1490 ts_tablet_manager.cc:1905] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:44.089550 1489 ts_tablet_manager.cc:1905] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:46:44.089277 1490 ts_tablet_manager.cc:1918] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:44.091246 1490 log.cc:1196] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/83001600c899496892772b466bd335c9
I20260810 04:46:44.091336 1489 ts_tablet_manager.cc:1918] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:46:44.091415 1489 log.cc:1196] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/83001600c899496892772b466bd335c9
I20260810 04:46:44.091741 1489 ts_tablet_manager.cc:1939] T 83001600c899496892772b466bd335c9 P eb1c66a9efa44d8f8629efe330c8a011: Deleting consensus metadata
I20260810 04:46:44.091768 1490 ts_tablet_manager.cc:1939] T 83001600c899496892772b466bd335c9 P 950a9fcafec340f190dbbd391b0a5abc: Deleting consensus metadata
I20260810 04:46:44.092242 656 catalog_manager.cc:4841] TS 950a9fcafec340f190dbbd391b0a5abc (127.14.237.2:41453): tablet 83001600c899496892772b466bd335c9 (table test-table [id=f835fa93e867448bb5356bcf5444ba07]) successfully deleted
I20260810 04:46:44.092379 656 catalog_manager.cc:4841] TS eb1c66a9efa44d8f8629efe330c8a011 (127.14.237.3:38137): tablet 83001600c899496892772b466bd335c9 (table test-table [id=f835fa93e867448bb5356bcf5444ba07]) successfully deleted
W20260810 04:46:44.097206 656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:34853 (error 108)
I20260810 04:46:44.097776 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:46:44.097610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:34853 (local address 127.0.0.1:38752)
0810 04:46:44.097676 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 04:46:44.097722 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 04:46:44.097837 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 1002
W20260810 04:46:44.113684 656 catalog_manager.cc:4584] TS 19b6e50b050b49d88ce8af7b760e75d0 (127.14.237.1:34853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83001600c899496892772b466bd335c9: Network error: Client connection negotiation failed: client connection to 127.14.237.1:34853: connect: Connection refused (error 111)
I20260810 04:46:44.118338 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 1217
I20260810 04:46:44.127290 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 632
2026-08-10T04:46:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2167 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:46:44 dist-test-slave-8hhm krb5kdc[1497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:46:44 dist-test-slave-8hhm krb5kdc[1497](info): set up 2 sockets
Aug 10 04:46:44 dist-test-slave-8hhm krb5kdc[1497](info): commencing operation
krb5kdc: starting...
W20260810 04:46:45.398772 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:46:45 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:46:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.237.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.237.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/rangeradmin_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/rangeradmin_127.14.237.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.237.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.237.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/rangerlookup_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/rangerlookup_127.14.237.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.237.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.237.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/HTTP_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/HTTP_127.14.237.20@KRBTEST.COM.keytab.
I20260810 04:46:45.565469 15284 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwUvI9j/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/postgres -l logfile start
2026-08-10 04:46:47.994 UTC [1547] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 04:46:47.994 UTC [1547] LOG: listening on IPv4 address "127.14.237.20", port 60247
2026-08-10 04:46:47.998 UTC [1547] LOG: listening on Unix socket "/tmp/.s.PGSQL.60247"
2026-08-10 04:46:48.033 UTC [1550] LOG: database system was shut down at 2026-08-10 04:46:46 UTC
2026-08-10 04:46:48.037 UTC [1547] LOG: database system is ready to accept connections
I20260810 04:46:49.196995 15284 mini_postgres.cc:96] Postgres bound to 60247
2026-08-10 04:46:49.205 UTC [1561] FATAL: database "slave" does not exist
127.14.237.20:60247 - accepting connections
I20260810 04:46:49.206643 15284 mini_ranger.cc:161] Starting Ranger...
I20260810 04:46:49.222734 15284 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 04:46:49.669667 15284 mini_ranger.cc:87] Created ranger Postgres database
I20260810 04:46:49.669749 15284 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-admin
2026-08-10 04:46:50,013 [I] DB FLAVOR :POSTGRES
2026-08-10 04:46:50,013 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:46:50,014 [I] Checking connection..
2026-08-10 04:46:50,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 04:46:50,276 [I] Checking connection passed.
2026-08-10 04:46:50,276 [I] --------- Verifying version history table ---------
2026-08-10 04:46:50,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:46:50,527 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:46:50,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:46:50,788 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:46:50,788 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 04:46:50,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 04:46:51.049 UTC [1670] WARNING: there is no transaction in progress
2026-08-10 04:46:51,056 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 04:46:51,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:46:51,312 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 04:46:51,312 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:46:51,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 04:46:51,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 04:46:51,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8hhm.c.gcp-upstream.internal','N') ;"
2026-08-10 04:46:52,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:46:52,324 [I] Table x_portal_user does not exist in database ranger
2026-08-10 04:46:52,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:46:52,576 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 04:46:52,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:46:52,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:46:53,098 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 04:46:53,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 04:46:54.012 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.029 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.045 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.059 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.074 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.089 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.141 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.147 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.161 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.171 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.181 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.193 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.203 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.209 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.216 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54.223 UTC [1868] WARNING: there is no transaction in progress
2026-08-10 04:46:54,839 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 04:46:54,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 04:46:55,100 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 04:46:55,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:46:55,356 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 04:46:55,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:46:55,633 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 04:46:55,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:46:55,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:46:56,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8hhm.c.gcp-upstream.internal';"
2026-08-10 04:46:56,380 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 04:46:56,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 04:46:56,639 [I] DB_PATCHES have already been applied
I20260810 04:46:56.645198 15284 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwUvI9j/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 04:46:56.645324 15284 mini_ranger.cc:193] Using host: 127.14.237.20
I20260810 04:46:56.649103 15284 mini_ranger.cc:227] Ranger admin URL: http://127.14.237.20:48831
Aug 10, 2026 4:46:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:46:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:46:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 4:46:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 4:46:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 4:46:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 4:46:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 04:46:58.039834 15284 mini_ranger.cc:160] Time spent starting Ranger: real 8.833s user 0.004s sys 0.007s
Aug 10, 2026 4:47:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 4:47:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 4:47:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 4:47:18 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 4:47:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 04:47:20.170976 15284 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.237.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.237.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/rangerkms_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/rangerkms_127.14.237.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.237.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.237.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.237.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/HTTP_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.237.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/HTTP_127.14.237.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 04:47:20 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337240, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 04:47:20.266991 15284 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 04:47:20.277755 15284 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 04:47:21.066617 15284 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 04:47:21.066711 15284 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms
I20260810 04:47:21.066730 15284 mini_ranger_kms.cc:227] Using postgres at 127.14.237.20:60247
2026-08-10 04:47:21,588 [I] DB FLAVOR :POSTGRES
2026-08-10 04:47:21,589 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:47:21,589 [I] Checking connection
2026-08-10 04:47:21,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 04:47:21,824 [I] connection success
2026-08-10 04:47:21,824 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 04:47:21,825 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 04:47:21,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 04:47:22,057 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 04:47:22,057 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:47:22,058 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 04:47:22,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:60247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 04:47:22,343 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 04:47:22.811450 15284 mini_ranger_kms.cc:310] Created kms service
I20260810 04:47:23.022497 15284 mini_ranger_kms.cc:326] Created kudu user
I20260810 04:47:23.066219 15284 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 04:47:23 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337240, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:47:23.451037 15284 mini_ranger_kms.cc:384] Added ranger policy
I20260810 04:47:23.451161 15284 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 04:47:23.451184 15284 mini_ranger_kms.cc:240] Using host: 127.14.237.20
I20260810 04:47:23.455565 15284 mini_ranger_kms.cc:276] Ranger KMS PID: 2181
I20260810 04:47:23.455693 15284 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.237.20:40809
Aug 10, 2026 4:47:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:47:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:47:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 4:47:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 4:47:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 4:47:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 04:47:25 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786337245, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.237.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 04:47:26 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337245, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.237.20@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:47:26.300244 15284 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.033s user 0.000s sys 0.255s
I20260810 04:47:26.300350 15284 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 04:47:26.300423 15284 mini_ranger_kms.cc:401] 127.14.237.20:40809/kms/v1/keys
Aug 10 04:47:26 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337245, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.237.20@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
Aug 10 04:47:28 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:47:28.812734 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38961
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:38961
--ranger_config_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:40809/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:47:28.896457 2271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:47:28.896646 2271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:47:28.896667 2271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:47:28.897958 2271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:47:28.898000 2271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:47:28.898015 2271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:47:28.898027 2271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:47:28.898065 2271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:47:28.899796 2271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.237.20:40809/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:38961
--ranger_config_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:47:28.900071 2271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:47:28.900308 2271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:47:28.904893 2278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:47:28.904909 2277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:47:28.905000 2282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:47:28.905310 2271 server_base.cc:1034] running on GCE node
I20260810 04:47:28.905570 2271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:47:28.905862 2271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:47:28.907018 2271 hybrid_clock.cc:648] HybridClock initialized: now 1786337248906993 us; error 41 us; skew 500 ppm
Aug 10 04:47:28 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:47:28.909822 2271 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:47:28.910326 2271 webserver.cc:460] Webserver started at http://127.14.237.62:40649/ using document root <none> and password file <none>
I20260810 04:47:28.910583 2271 fs_manager.cc:359] Metadata directory not provided
I20260810 04:47:28.910634 2271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:47:28.910768 2271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:47:28 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:47:28.962109 2271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "44ba5d8a33b742f8996c1033adcc8e72"
format_stamp: "Formatted at 2026-08-10 04:47:28 on dist-test-slave-8hhm"
server_key: "51eb299f1fe28c60ba2d15fd9350daf6"
server_key_iv: "d30e8b7945026f946ec3ff557c9c5cb8"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:28.962635 2271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "44ba5d8a33b742f8996c1033adcc8e72"
format_stamp: "Formatted at 2026-08-10 04:47:28 on dist-test-slave-8hhm"
server_key: "51eb299f1fe28c60ba2d15fd9350daf6"
server_key_iv: "d30e8b7945026f946ec3ff557c9c5cb8"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:28.964555 2271 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 04:47:28.978029 2292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:28.978250 2271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:47:28.978353 2271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "44ba5d8a33b742f8996c1033adcc8e72"
format_stamp: "Formatted at 2026-08-10 04:47:28 on dist-test-slave-8hhm"
server_key: "51eb299f1fe28c60ba2d15fd9350daf6"
server_key_iv: "d30e8b7945026f946ec3ff557c9c5cb8"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:28.978437 2271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:47:28.998131 2271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:47:29.000070 2271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:47:29.000224 2271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:47:29.005499 2271 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:38961
I20260810 04:47:29.005527 2376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:38961 every 8 connection(s)
I20260810 04:47:29.005959 2271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:47:29.006762 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:47:29.009112 2378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72: Bootstrap starting.
I20260810 04:47:29.009750 2378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72: Neither blocks nor log segments found. Creating new log.
I20260810 04:47:29.010025 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 2271
I20260810 04:47:29.010146 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:47:29.010197 2378 log.cc:824] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72: Log is configured to *not* fsync() on all Append() calls
I20260810 04:47:29.010353 15284 external_mini_cluster.cc:1428] Setting key 7bc103b535c8a64a90073fd7b97af0dc
I20260810 04:47:29.011209 2378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72: No bootstrap required, opened a new log
I20260810 04:47:29.013876 2378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ba5d8a33b742f8996c1033adcc8e72" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38961 } }
I20260810 04:47:29.014037 2378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:47:29.014066 2378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ba5d8a33b742f8996c1033adcc8e72, State: Initialized, Role: FOLLOWER
I20260810 04:47:29.014225 2378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ba5d8a33b742f8996c1033adcc8e72" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38961 } }
I20260810 04:47:29.014314 2378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:47:29.014384 2378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:47:29.014451 2378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:47:29.015162 2378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ba5d8a33b742f8996c1033adcc8e72" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38961 } }
I20260810 04:47:29.015275 2378 leader_election.cc:302] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44ba5d8a33b742f8996c1033adcc8e72; no voters:
I20260810 04:47:29.015473 2378 leader_election.cc:288] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:47:29.015518 2383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:47:29.015736 2383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [term 1 LEADER]: Becoming Leader. State: Replica: 44ba5d8a33b742f8996c1033adcc8e72, State: Running, Role: LEADER
I20260810 04:47:29.015890 2378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:47:29.015890 2383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ba5d8a33b742f8996c1033adcc8e72" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38961 } }
I20260810 04:47:29.017395 2384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44ba5d8a33b742f8996c1033adcc8e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ba5d8a33b742f8996c1033adcc8e72" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38961 } } }
I20260810 04:47:29.017584 2384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [sys.catalog]: This master's current role is: LEADER
I20260810 04:47:29.017664 2378 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 04:47:29.017887 2385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ba5d8a33b742f8996c1033adcc8e72. Latest consensus state: current_term: 1 leader_uuid: "44ba5d8a33b742f8996c1033adcc8e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ba5d8a33b742f8996c1033adcc8e72" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38961 } } }
I20260810 04:47:29.018085 2385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72 [sys.catalog]: This master's current role is: LEADER
I20260810 04:47:29.018289 2404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 04:47:29 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:47:29.018841 2404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:47:29.027545 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:29.011011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.14.237.62:38961)
0810 04:47:29.011195 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:29.011204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:29.011236 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:47:29.011738 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:29.011748 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:29.011822 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:29.011976 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:47:29.012786 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:29.014254 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:29.015402 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:29.016068 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:29.016188 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:29.020700 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:47:29.020764 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:47:29.020764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:47:29.020858 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:47:29.025264 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:29.025733 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:29.025736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:29.025737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:29.025923 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:29.026134 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:29.026134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:29.026135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:29.026456 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:47:29.027110 (+ 654us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:29.027185 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 04:47:29.027326 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260810 04:47:29.029824 2404 catalog_manager.cc:1347] Generated new cluster ID: 6226a6d5e6d74d3da3638c15b5c8726c
I20260810 04:47:29.029884 2404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:47:29.064561 2404 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:47:29.065613 2404 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:47:29.070073 2404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72: Generated new TSK 0
I20260810 04:47:29.070349 2404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:47:29.830837 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:40809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38961
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:47:29.955281 2452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:47:29.955632 2452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:47:29.955786 2452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:47:29.957440 2452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:47:29.957545 2452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:47:29.957633 2452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:47:29.959771 2452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.237.20:40809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:47:29.960276 2452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:47:29.960623 2452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:47:29.966894 2461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:47:29.967478 2460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:47:29.967506 2463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:47:29.967788 2452 server_base.cc:1034] running on GCE node
I20260810 04:47:29.968258 2452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:47:29.968511 2452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:47:29.969733 2452 hybrid_clock.cc:648] HybridClock initialized: now 1786337249969704 us; error 41 us; skew 500 ppm
Aug 10 04:47:29 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:47:29.977119 2452 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:47:29.977658 2452 webserver.cc:460] Webserver started at http://127.14.237.1:41407/ using document root <none> and password file <none>
I20260810 04:47:29.977933 2452 fs_manager.cc:359] Metadata directory not provided
I20260810 04:47:29.977990 2452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:47:29.978121 2452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:47:29 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:47:29.991585 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "d2ab77159c784113bc55cef5c8b2b24d"
format_stamp: "Formatted at 2026-08-10 04:47:29 on dist-test-slave-8hhm"
server_key: "ce9e623a941e4efd7c6b7edb085f5ecc"
server_key_iv: "93df610de52e0513288ba1ca546178b2"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:29.992274 2452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "d2ab77159c784113bc55cef5c8b2b24d"
format_stamp: "Formatted at 2026-08-10 04:47:29 on dist-test-slave-8hhm"
server_key: "ce9e623a941e4efd7c6b7edb085f5ecc"
server_key_iv: "93df610de52e0513288ba1ca546178b2"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:29.994098 2452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:47:30.010227 2476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.010456 2452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:47:30.010545 2452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "d2ab77159c784113bc55cef5c8b2b24d"
format_stamp: "Formatted at 2026-08-10 04:47:29 on dist-test-slave-8hhm"
server_key: "ce9e623a941e4efd7c6b7edb085f5ecc"
server_key_iv: "93df610de52e0513288ba1ca546178b2"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:30.010733 2452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:47:30.031857 2452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:47:30.034068 2452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:47:30.034246 2452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:47:30.034483 2452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:47:30.034868 2452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:47:30.034919 2452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.034986 2452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:47:30.035029 2452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.041899 2452 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:46161
I20260810 04:47:30.042470 2452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:47:30.045435 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 2452
I20260810 04:47:30.045534 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:47:30.045722 15284 external_mini_cluster.cc:1428] Setting key e4b44810be3464d7564154f1227574e6
I20260810 04:47:30.045852 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:46161 every 8 connection(s)
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:47:30.061844 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.046878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38961 (local address 127.14.237.1:49093)
0810 04:47:30.051837 (+ 4959us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.051883 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:47:30.052475 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:47:30.052823 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:47:30.052827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:47:30.053085 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:47:30.053397 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.053404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.054370 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.054371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.055154 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.055156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.055665 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.055666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.055768 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.056382 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:47:30.056391 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:47:30.057941 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:47:30.060088 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:47:30.060095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:47:30.060097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:47:30.060415 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:47:30.060660 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:47:30.060661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:47:30.060661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:47:30.060799 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:47:30.061108 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:47:30.061115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:47:30.061305 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 04:47:30.061413 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 04:47:30.061572 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4890,"thread_start_us":89,"threads_started":1}
I20260810 04:47:30.062007 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.047195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:49093 (local address 127.14.237.62:38961)
0810 04:47:30.051739 (+ 4544us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.051741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.052529 (+ 788us) server_negotiation.cc:407] Connection header received
0810 04:47:30.052563 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.052564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.052578 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.052676 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:47:30.053518 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.054281 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.055257 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.055558 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.055648 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.058020 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:47:30.058021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:47:30.058022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:47:30.058029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:47:30.059980 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:30.060506 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:30.060507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:30.060507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:30.060572 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:30.060874 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:30.060874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:30.060875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:30.061017 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:47:30.061081 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.061796 (+ 715us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.061893 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4513,"thread_start_us":77,"threads_started":1}
I20260810 04:47:30.062857 2671 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38961
I20260810 04:47:30.062959 2671 heartbeater.cc:464] Registering TS with master...
I20260810 04:47:30.063139 2671 heartbeater.cc:511] Master 127.14.237.62:38961 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
I20260810 04:47:30.063702 2321 ts_manager.cc:194] Registered new tserver with Master: d2ab77159c784113bc55cef5c8b2b24d (127.14.237.1:46161)
I20260810 04:47:30.064391 2321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:49093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:47:30.082664 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:40809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38961
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786337250, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 04:47:30.203270 2686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:47:30.203584 2686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:47:30.203629 2686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:47:30.205272 2686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:47:30.205384 2686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:47:30.205453 2686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:47:30.207623 2686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.237.20:40809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:47:30.207993 2686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:47:30.208280 2686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:47:30.214794 2701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:47:30.215183 2703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:47:30.215839 2699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:47:30.216272 2686 server_base.cc:1034] running on GCE node
I20260810 04:47:30.216464 2686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:47:30.216714 2686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:47:30.217962 2686 hybrid_clock.cc:648] HybridClock initialized: now 1786337250217903 us; error 69 us; skew 500 ppm
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337250, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:47:30.221163 2686 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:47:30.221681 2686 webserver.cc:460] Webserver started at http://127.14.237.2:33243/ using document root <none> and password file <none>
I20260810 04:47:30.221976 2686 fs_manager.cc:359] Metadata directory not provided
I20260810 04:47:30.222071 2686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:47:30.222262 2686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337250, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:47:30.247809 2686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "3bd9e16f565a4826b142c70704be9460"
format_stamp: "Formatted at 2026-08-10 04:47:30 on dist-test-slave-8hhm"
server_key: "b1221f096f44a54bd00d179cbd096664"
server_key_iv: "d48768bfac9b3433f0e024c6369b5209"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:30.248288 2686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "3bd9e16f565a4826b142c70704be9460"
format_stamp: "Formatted at 2026-08-10 04:47:30 on dist-test-slave-8hhm"
server_key: "b1221f096f44a54bd00d179cbd096664"
server_key_iv: "d48768bfac9b3433f0e024c6369b5209"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:30.250793 2686 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 04:47:30.268760 2718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.269016 2686 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260810 04:47:30.269130 2686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "3bd9e16f565a4826b142c70704be9460"
format_stamp: "Formatted at 2026-08-10 04:47:30 on dist-test-slave-8hhm"
server_key: "b1221f096f44a54bd00d179cbd096664"
server_key_iv: "d48768bfac9b3433f0e024c6369b5209"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:30.269320 2686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:47:30.283677 2686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:47:30.286425 2686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:47:30.286576 2686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:47:30.286787 2686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:47:30.287145 2686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:47:30.287197 2686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.287258 2686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:47:30.287297 2686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.295195 2686 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:39879
I20260810 04:47:30.295565 2686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:47:30.295990 2900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:39879 every 8 connection(s)
I20260810 04:47:30.301057 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 2686
I20260810 04:47:30.301174 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:47:30.301505 15284 external_mini_cluster.cc:1428] Setting key 9b083523456e8f61fa273db697234c4e
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337250, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:47:30.309754 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.296640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:36081 (local address 127.14.237.62:38961)
0810 04:47:30.296684 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.296687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.296698 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:47:30.297190 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.297191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.297226 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.297281 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:47:30.297902 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.298658 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.300116 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.300420 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.300463 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.306422 (+ 5959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:47:30.306424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:47:30.306424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:47:30.306431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:47:30.308103 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:30.308599 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:30.308600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:30.308601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:30.308680 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:30.308970 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:30.308971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:30.308972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:30.309101 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:47:30.309144 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.309552 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.309662 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:47:30.310496 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.296352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38961 (local address 127.14.237.2:36081)
0810 04:47:30.296496 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.296539 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:47:30.297073 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:47:30.297358 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:47:30.297361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:47:30.297604 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:47:30.297802 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.297806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.298780 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.298781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.300008 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.300011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.300555 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.300556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.300717 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.304522 (+ 3805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:47:30.304533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:47:30.306301 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:47:30.308203 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:47:30.308210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:47:30.308212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:47:30.308501 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:47:30.308755 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:47:30.308756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:47:30.308756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:47:30.308886 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:47:30.309193 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:47:30.309199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:47:30.309384 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 04:47:30.310104 (+ 720us) client_negotiation.cc:240] Negotiation successful
0810 04:47:30.310270 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260810 04:47:30.311162 2901 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38961
I20260810 04:47:30.311309 2901 heartbeater.cc:464] Registering TS with master...
I20260810 04:47:30.311549 2901 heartbeater.cc:511] Master 127.14.237.62:38961 requested a full tablet report, sending...
I20260810 04:47:30.312011 2321 ts_manager.cc:194] Registered new tserver with Master: 3bd9e16f565a4826b142c70704be9460 (127.14.237.2:39879)
I20260810 04:47:30.312470 2321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:36081
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:47:30.350381 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:40809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38961
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:47:30.454548 2908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:47:30.454818 2908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:47:30.454849 2908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:47:30.456496 2908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:47:30.456553 2908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:47:30.456593 2908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:47:30.458606 2908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.237.20:40809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:47:30.458840 2908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:47:30.459082 2908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:47:30.464601 2921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:47:30.464648 2918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:47:30.464859 2917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:47:30.464970 2908 server_base.cc:1034] running on GCE node
I20260810 04:47:30.465170 2908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:47:30.465427 2908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:47:30.466596 2908 hybrid_clock.cc:648] HybridClock initialized: now 1786337250466582 us; error 29 us; skew 500 ppm
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337250, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:47:30.469082 2908 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:47:30.469506 2908 webserver.cc:460] Webserver started at http://127.14.237.3:40983/ using document root <none> and password file <none>
I20260810 04:47:30.469741 2908 fs_manager.cc:359] Metadata directory not provided
I20260810 04:47:30.469808 2908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:47:30.469955 2908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337250, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:47:30.483844 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "349427b85cc44ae9b80f06562d6325ee"
format_stamp: "Formatted at 2026-08-10 04:47:30 on dist-test-slave-8hhm"
server_key: "2c2aa83deaaef8455f6d9964cd58df71"
server_key_iv: "67f4aa4560f7296fb2a959a250814eac"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:30.484397 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "349427b85cc44ae9b80f06562d6325ee"
format_stamp: "Formatted at 2026-08-10 04:47:30 on dist-test-slave-8hhm"
server_key: "2c2aa83deaaef8455f6d9964cd58df71"
server_key_iv: "67f4aa4560f7296fb2a959a250814eac"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:30.491379 2908 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260810 04:47:30.508721 2935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.508970 2908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 04:47:30.509065 2908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "349427b85cc44ae9b80f06562d6325ee"
format_stamp: "Formatted at 2026-08-10 04:47:30 on dist-test-slave-8hhm"
server_key: "2c2aa83deaaef8455f6d9964cd58df71"
server_key_iv: "67f4aa4560f7296fb2a959a250814eac"
server_key_version: "kuduclusterkey@0"
I20260810 04:47:30.509171 2908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:47:30.526829 2908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:47:30.529497 2908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:47:30.529739 2908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:47:30.530031 2908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:47:30.530478 2908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:47:30.530586 2908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.530674 2908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:47:30.530736 2908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:47:30.537719 2908 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:42131
I20260810 04:47:30.538213 2908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:47:30.541086 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 2908
I20260810 04:47:30.541193 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:47:30.541456 15284 external_mini_cluster.cc:1428] Setting key 06008217c084d26f7547b34ee772f55b
I20260810 04:47:30.548502 3126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:42131 every 8 connection(s)
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337250, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:47:30.561167 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.549575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:40619 (local address 127.14.237.62:38961)
0810 04:47:30.549632 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.549634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.550084 (+ 450us) server_negotiation.cc:407] Connection header received
0810 04:47:30.550811 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.550813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.550827 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.550860 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:47:30.551814 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.552558 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.554669 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.554948 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.554998 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.557676 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:47:30.557677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:47:30.557678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:47:30.557686 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:47:30.559481 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:30.560010 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:30.560011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:30.560012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:30.560089 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:30.560384 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:30.560385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:30.560385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:30.560525 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:47:30.560586 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.560983 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.561069 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:47:30.562047 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.549190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38961 (local address 127.14.237.3:40619)
0810 04:47:30.549942 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.549997 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 04:47:30.550685 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:47:30.551054 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:47:30.551059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:47:30.551338 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:47:30.551684 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.551691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.553275 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.553276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.554549 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.554552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.555097 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.555098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.555239 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.555834 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:47:30.555843 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:47:30.557548 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:47:30.559581 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:47:30.559587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:47:30.559590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:47:30.559905 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:47:30.560161 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:47:30.560162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:47:30.560162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:47:30.560297 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:47:30.560640 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:47:30.560647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:47:30.560811 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 04:47:30.561642 (+ 831us) client_negotiation.cc:240] Negotiation successful
0810 04:47:30.561811 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":186,"threads_started":1}
I20260810 04:47:30.562848 3127 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38961
I20260810 04:47:30.562945 3127 heartbeater.cc:464] Registering TS with master...
I20260810 04:47:30.563140 3127 heartbeater.cc:511] Master 127.14.237.62:38961 requested a full tablet report, sending...
I20260810 04:47:30.564177 2321 ts_manager.cc:194] Registered new tserver with Master: 349427b85cc44ae9b80f06562d6325ee (127.14.237.3:42131)
I20260810 04:47:30.564656 2321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:40619
I20260810 04:47:30.574621 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:47:30.584695 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.576380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.14.237.62:38961)
0810 04:47:30.576422 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.576424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.576747 (+ 323us) server_negotiation.cc:407] Connection header received
0810 04:47:30.576844 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.576845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.576856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.576885 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:47:30.577744 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.578411 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.579441 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.579740 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.579778 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.581578 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:47:30.581579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:47:30.581580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:47:30.581587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:47:30.583283 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:30.583779 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:30.583780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:30.583780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:30.583846 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:47:30.584109 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:47:30.584111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:47:30.584111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:47:30.584251 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:47:30.584355 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.584525 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.584608 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:47:30.586560 2321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:47:30.588876 2321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:47:30.597589 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.592488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:39879 (local address 127.0.0.1:54308)
0810 04:47:30.593472 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.593481 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:47:30.593525 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:47:30.593809 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:47:30.593812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:47:30.593815 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:47:30.593873 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.593875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.595123 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.595124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.596292 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.596294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.597296 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.597298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.597378 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.597392 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:47:30.597461 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:47:30.597511 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":65,"threads_started":1}
I20260810 04:47:30.597983 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.592847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.14.237.2:39879)
0810 04:47:30.593152 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.593156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.593580 (+ 424us) server_negotiation.cc:407] Connection header received
0810 04:47:30.593625 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.593627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.593643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.593715 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:47:30.593959 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.595026 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.596389 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.597071 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.597152 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.597258 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.597866 (+ 608us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.597905 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":75,"threads_started":1}
I20260810 04:47:30.598870 2797 tablet_service.cc:1467] Processing CreateTablet for tablet 33ba03d5300f438f9ef811d25fb77b76 (DEFAULT_TABLE table=test-table [id=e9e1c5d365f546d6ad310a92e5e8ad62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:47:30.599283 2797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ba03d5300f438f9ef811d25fb77b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:47:30.604887 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.592284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:46161 (local address 127.0.0.1:33516)
0810 04:47:30.593235 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.593278 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:47:30.593351 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:47:30.594042 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:47:30.594043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:47:30.594045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:47:30.594072 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.594073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.597591 (+ 3518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.597591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.603625 (+ 6034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.603628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.604657 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.604658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.604726 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.604731 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:47:30.604780 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:47:30.604818 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":68,"threads_started":1}
I20260810 04:47:30.605125 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.592708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.14.237.1:46161)
0810 04:47:30.593058 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.593063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.593431 (+ 368us) server_negotiation.cc:407] Connection header received
0810 04:47:30.593469 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.593471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.593486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.593558 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:47:30.594186 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.595231 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.603729 (+ 8498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.604488 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.604564 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.604618 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.605010 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.605052 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":57,"threads_started":1}
I20260810 04:47:30.605722 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.592305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:42131 (local address 127.0.0.1:35348)
0810 04:47:30.593323 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.593335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:47:30.593392 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:47:30.593922 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:47:30.593923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:47:30.593925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:47:30.593970 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.593971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.596420 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.596421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.597499 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.597502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.604697 (+ 7195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.604698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.605547 (+ 849us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.605552 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:47:30.605611 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:47:30.605654 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":74,"threads_started":1}
I20260810 04:47:30.605960 2525 tablet_service.cc:1467] Processing CreateTablet for tablet 33ba03d5300f438f9ef811d25fb77b76 (DEFAULT_TABLE table=test-table [id=e9e1c5d365f546d6ad310a92e5e8ad62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:47:30.606184 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.592539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.14.237.3:42131)
0810 04:47:30.593693 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.593697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.593707 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:47:30.593744 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.593746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.593761 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.593841 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:47:30.595332 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.596328 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.603759 (+ 7431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.604513 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.604587 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.604660 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.606039 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.606082 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":75,"threads_started":1}
I20260810 04:47:30.606326 2525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ba03d5300f438f9ef811d25fb77b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:47:30.606773 2982 tablet_service.cc:1467] Processing CreateTablet for tablet 33ba03d5300f438f9ef811d25fb77b76 (DEFAULT_TABLE table=test-table [id=e9e1c5d365f546d6ad310a92e5e8ad62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:47:30.607087 2982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ba03d5300f438f9ef811d25fb77b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:47:30.610795 3158 tablet_bootstrap.cc:492] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: Bootstrap starting.
I20260810 04:47:30.611078 3155 tablet_bootstrap.cc:492] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460: Bootstrap starting.
I20260810 04:47:30.611892 3158 tablet_bootstrap.cc:654] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: Neither blocks nor log segments found. Creating new log.
I20260810 04:47:30.612236 3155 tablet_bootstrap.cc:654] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460: Neither blocks nor log segments found. Creating new log.
I20260810 04:47:30.612251 3158 log.cc:824] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: Log is configured to *not* fsync() on all Append() calls
I20260810 04:47:30.612705 3155 log.cc:824] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460: Log is configured to *not* fsync() on all Append() calls
I20260810 04:47:30.613601 3155 tablet_bootstrap.cc:492] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460: No bootstrap required, opened a new log
I20260810 04:47:30.613672 3155 ts_tablet_manager.cc:1397] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:47:30.616119 3155 raft_consensus.cc:348] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.616313 3155 raft_consensus.cc:374] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:47:30.616384 3155 raft_consensus.cc:729] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd9e16f565a4826b142c70704be9460, State: Initialized, Role: FOLLOWER
I20260810 04:47:30.616528 3155 consensus_queue.cc:260] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.616936 2901 heartbeater.cc:503] Master 127.14.237.62:38961 was elected leader, sending a full tablet report...
I20260810 04:47:30.617290 3155 ts_tablet_manager.cc:1428] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 04:47:30.621411 3156 tablet_bootstrap.cc:492] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d: Bootstrap starting.
I20260810 04:47:30.622076 3156 tablet_bootstrap.cc:654] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d: Neither blocks nor log segments found. Creating new log.
I20260810 04:47:30.622427 3156 log.cc:824] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:47:30.622510 3158 tablet_bootstrap.cc:492] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: No bootstrap required, opened a new log
I20260810 04:47:30.622586 3158 ts_tablet_manager.cc:1397] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260810 04:47:30.624677 3158 raft_consensus.cc:348] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.624848 3158 raft_consensus.cc:374] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:47:30.624882 3158 raft_consensus.cc:729] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349427b85cc44ae9b80f06562d6325ee, State: Initialized, Role: FOLLOWER
I20260810 04:47:30.624996 3158 consensus_queue.cc:260] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.625303 3158 ts_tablet_manager.cc:1428] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:47:30.625535 3127 heartbeater.cc:503] Master 127.14.237.62:38961 was elected leader, sending a full tablet report...
I20260810 04:47:30.625589 3156 tablet_bootstrap.cc:492] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d: No bootstrap required, opened a new log
I20260810 04:47:30.625644 3156 ts_tablet_manager.cc:1397] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:47:30.628062 3156 raft_consensus.cc:348] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.628278 3156 raft_consensus.cc:374] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:47:30.628355 3156 raft_consensus.cc:729] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ab77159c784113bc55cef5c8b2b24d, State: Initialized, Role: FOLLOWER
I20260810 04:47:30.628521 3156 consensus_queue.cc:260] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.628798 3156 ts_tablet_manager.cc:1428] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:47:30.628942 2671 heartbeater.cc:503] Master 127.14.237.62:38961 was elected leader, sending a full tablet report...
Aug 10 04:47:30 dist-test-slave-8hhm krb5kdc[1497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337250, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:47:30.729558 3169 raft_consensus.cc:482] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:47:30.729710 3169 raft_consensus.cc:504] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.730134 3169 leader_election.cc:288] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2ab77159c784113bc55cef5c8b2b24d (127.14.237.1:46161), 3bd9e16f565a4826b142c70704be9460 (127.14.237.2:39879)
I20260810 04:47:30.734272 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.730720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:54833 (local address 127.14.237.2:39879)
0810 04:47:30.730762 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.730765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.730773 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:47:30.730802 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.730803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.730814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.730870 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:47:30.731083 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.731877 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.733376 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.733909 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.733953 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.733962 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.734165 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.734201 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:47:30.734305 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.730326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:46161 (local address 127.14.237.3:35259)
0810 04:47:30.730558 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.730570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:47:30.730636 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:47:30.731040 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:47:30.731040 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:47:30.731042 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:47:30.731109 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.731111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.732038 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.732039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.733273 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.733276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.734042 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.734043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.734122 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.734125 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:47:30.734169 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 04:47:30.734220 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1}
I20260810 04:47:30.734653 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.730440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:39879 (local address 127.14.237.3:54833)
0810 04:47:30.730623 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.730633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:47:30.730703 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:47:30.730892 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:47:30.730893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:47:30.730895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:47:30.730985 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.730986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.731997 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.731998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.733279 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:47:30.733282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.734018 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.734018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:47:30.734090 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.734094 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:47:30.734518 (+ 424us) client_negotiation.cc:240] Negotiation successful
0810 04:47:30.734561 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":43,"thread_start_us":115,"threads_started":1}
I20260810 04:47:30.734748 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.730400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:35259 (local address 127.14.237.1:46161)
0810 04:47:30.730475 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.730478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.730911 (+ 433us) server_negotiation.cc:407] Connection header received
0810 04:47:30.730939 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.730939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.730952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.730996 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:47:30.731178 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.731935 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.733373 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.733932 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.734003 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.734010 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.734641 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.734684 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 04:47:30.735145 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ba03d5300f438f9ef811d25fb77b76" candidate_uuid: "349427b85cc44ae9b80f06562d6325ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ab77159c784113bc55cef5c8b2b24d" is_pre_election: true
I20260810 04:47:30.735143 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ba03d5300f438f9ef811d25fb77b76" candidate_uuid: "349427b85cc44ae9b80f06562d6325ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd9e16f565a4826b142c70704be9460" is_pre_election: true
I20260810 04:47:30.735311 2827 raft_consensus.cc:2454] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349427b85cc44ae9b80f06562d6325ee in term 0.
I20260810 04:47:30.735311 2588 raft_consensus.cc:2454] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 349427b85cc44ae9b80f06562d6325ee in term 0.
I20260810 04:47:30.735580 2936 leader_election.cc:302] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349427b85cc44ae9b80f06562d6325ee, 3bd9e16f565a4826b142c70704be9460; no voters:
I20260810 04:47:30.735729 3169 raft_consensus.cc:2781] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:47:30.735798 3169 raft_consensus.cc:482] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:47:30.735834 3169 raft_consensus.cc:3037] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:47:30.736521 3169 raft_consensus.cc:504] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.736649 3169 leader_election.cc:288] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [CANDIDATE]: Term 1 election: Requested vote from peers d2ab77159c784113bc55cef5c8b2b24d (127.14.237.1:46161), 3bd9e16f565a4826b142c70704be9460 (127.14.237.2:39879)
I20260810 04:47:30.736811 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ba03d5300f438f9ef811d25fb77b76" candidate_uuid: "349427b85cc44ae9b80f06562d6325ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ab77159c784113bc55cef5c8b2b24d"
I20260810 04:47:30.736831 2827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ba03d5300f438f9ef811d25fb77b76" candidate_uuid: "349427b85cc44ae9b80f06562d6325ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd9e16f565a4826b142c70704be9460"
I20260810 04:47:30.736896 2827 raft_consensus.cc:3037] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:47:30.736889 2588 raft_consensus.cc:3037] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:47:30.737774 2827 raft_consensus.cc:2454] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349427b85cc44ae9b80f06562d6325ee in term 1.
I20260810 04:47:30.737856 2588 raft_consensus.cc:2454] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 349427b85cc44ae9b80f06562d6325ee in term 1.
I20260810 04:47:30.737941 2936 leader_election.cc:302] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 349427b85cc44ae9b80f06562d6325ee, 3bd9e16f565a4826b142c70704be9460; no voters:
I20260810 04:47:30.738083 3169 raft_consensus.cc:2781] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:47:30.738227 3169 raft_consensus.cc:686] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 1 LEADER]: Becoming Leader. State: Replica: 349427b85cc44ae9b80f06562d6325ee, State: Running, Role: LEADER
I20260810 04:47:30.738312 3169 consensus_queue.cc:237] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } }
I20260810 04:47:30.739204 2321 catalog_manager.cc:5495] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 349427b85cc44ae9b80f06562d6325ee (127.14.237.3). New cstate: current_term: 1 leader_uuid: "349427b85cc44ae9b80f06562d6325ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "349427b85cc44ae9b80f06562d6325ee" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 42131 } health_report { overall_health: HEALTHY } } }
I20260810 04:47:30.780078 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.777266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.14.237.3:42131)
0810 04:47:30.777329 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:47:30.777331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:47:30.777340 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:47:30.777385 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:47:30.777386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:47:30.777406 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:47:30.777492 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:47:30.777674 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.778151 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.779174 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:47:30.779433 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:47:30.779514 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:47:30.779698 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:47:30.779812 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:47:30.779902 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:47:30.779945 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:47:30.779966 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:47:30.782109 2827 raft_consensus.cc:1261] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 1 FOLLOWER]: Refusing update from remote peer 349427b85cc44ae9b80f06562d6325ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:47:30.782109 2588 raft_consensus.cc:1261] T 33ba03d5300f438f9ef811d25fb77b76 P d2ab77159c784113bc55cef5c8b2b24d [term 1 FOLLOWER]: Refusing update from remote peer 349427b85cc44ae9b80f06562d6325ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:47:30.782518 3174 consensus_queue.cc:1035] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ab77159c784113bc55cef5c8b2b24d" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 46161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:47:30.782627 3169 consensus_queue.cc:1035] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bd9e16f565a4826b142c70704be9460" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:47:30.786540 3179 mvcc.cc:205] Tried to move back new op lower bound from 7316837379200450560 to 7316837379024814080. Current Snapshot: MvccSnapshot[applied={T|T < 7316837379200450560}]
I20260810 04:47:30.786715 3180 mvcc.cc:205] Tried to move back new op lower bound from 7316837379200450560 to 7316837379024814080. Current Snapshot: MvccSnapshot[applied={T|T < 7316837379200450560}]
I20260810 04:47:30.786751 3181 mvcc.cc:205] Tried to move back new op lower bound from 7316837379200450560 to 7316837379024814080. Current Snapshot: MvccSnapshot[applied={T|T < 7316837379200450560}]
I20260810 04:47:30.789490 2321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:47:30.789595 2321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:47:30.790392 2321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44ba5d8a33b742f8996c1033adcc8e72: Sending DeleteTablet for 3 replicas of tablet 33ba03d5300f438f9ef811d25fb77b76
I20260810 04:47:30.790789 2982 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ba03d5300f438f9ef811d25fb77b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:47:30 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:35348
I20260810 04:47:30.790863 2525 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ba03d5300f438f9ef811d25fb77b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:47:30 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:33516
I20260810 04:47:30.791075 2797 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ba03d5300f438f9ef811d25fb77b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:47:30 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:54308
I20260810 04:47:30.791328 3196 tablet_replica.cc:330] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460: stopping tablet replica
I20260810 04:47:30.791472 3196 raft_consensus.cc:2229] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:47:30.791652 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 2452
I20260810 04:47:30.800940 3194 tablet_replica.cc:330] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: stopping tablet replica
I20260810 04:47:30.801098 3194 raft_consensus.cc:2229] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:47:30.801303 3194 raft_consensus.cc:2258] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:47:30.802604 3196 raft_consensus.cc:2258] T 33ba03d5300f438f9ef811d25fb77b76 P 3bd9e16f565a4826b142c70704be9460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:47:30.803042 3194 ts_tablet_manager.cc:1905] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:47:30.804713 3194 ts_tablet_manager.cc:1918] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:47:30.804869 3194 log.cc:1196] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/33ba03d5300f438f9ef811d25fb77b76
I20260810 04:47:30.805289 3194 ts_tablet_manager.cc:1939] T 33ba03d5300f438f9ef811d25fb77b76 P 349427b85cc44ae9b80f06562d6325ee: Deleting consensus metadata
I20260810 04:47:30.805764 2301 catalog_manager.cc:4841] TS 349427b85cc44ae9b80f06562d6325ee (127.14.237.3:42131): tablet 33ba03d5300f438f9ef811d25fb77b76 (table test-table [id=e9e1c5d365f546d6ad310a92e5e8ad62]) successfully deleted
W20260810 04:47:30.809273 2294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:46161 (error 108)
I20260810 04:47:30.810019 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 2686
I20260810 04:47:30.819020 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.818884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:46161 (local address 127.0.0.1:33532)
0810 04:47:30.818944 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.818982 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:46161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 04:47:30.819232 2294 catalog_manager.cc:4584] TS d2ab77159c784113bc55cef5c8b2b24d (127.14.237.1:46161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33ba03d5300f438f9ef811d25fb77b76: Network error: Client connection negotiation failed: client connection to 127.14.237.1:46161: connect: Connection refused (error 111)
I20260810 04:47:30.828475 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 2908
I20260810 04:47:30.830101 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:47:30.828551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:39879 (local address 127.0.0.1:54314)
0810 04:47:30.830005 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0810 04:47:30.830058 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.2:39879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1426}
I20260810 04:47:30.836962 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 2271
I20260810 04:47:30.847069 15284 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 04:47:31.191968 15284 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 04:47:31.192085 15284 mini_ranger.cc:66] Stopping Ranger...
I20260810 04:47:31.537714 15284 mini_ranger.cc:68] Stopped Ranger
2026-08-10 04:47:31.538 UTC [1547] LOG: received smart shutdown request
2026-08-10 04:47:31.543 UTC [1547] LOG: background worker "logical replication launcher" (PID 1557) exited with exit code 1
2026-08-10 04:47:31.544 UTC [1551] LOG: shutting down
2026-08-10 04:47:31.702 UTC [1547] LOG: database system is shut down
2026-08-10T04:47:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47685 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:47:31 dist-test-slave-8hhm krb5kdc[3214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:47:31 dist-test-slave-8hhm krb5kdc[3214](info): set up 2 sockets
Aug 10 04:47:31 dist-test-slave-8hhm krb5kdc[3214](info): commencing operation
krb5kdc: starting...
W20260810 04:47:33.066828 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:47:33 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:47:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.237.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.237.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/rangeradmin_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/rangeradmin_127.14.237.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.237.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.237.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/rangerlookup_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/rangerlookup_127.14.237.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.237.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.237.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/HTTP_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/HTTP_127.14.237.20@KRBTEST.COM.keytab.
I20260810 04:47:33.226007 15284 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwUvI9j/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/postgres -l logfile start
2026-08-10 04:47:35.556 UTC [3264] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 04:47:35.556 UTC [3264] LOG: listening on IPv4 address "127.14.237.20", port 41147
2026-08-10 04:47:35.559 UTC [3264] LOG: listening on Unix socket "/tmp/.s.PGSQL.41147"
2026-08-10 04:47:35.582 UTC [3268] LOG: database system was shut down at 2026-08-10 04:47:34 UTC
2026-08-10 04:47:35.586 UTC [3264] LOG: database system is ready to accept connections
I20260810 04:47:36.748219 15284 mini_postgres.cc:96] Postgres bound to 41147
2026-08-10 04:47:36.755 UTC [3280] FATAL: database "slave" does not exist
127.14.237.20:41147 - accepting connections
I20260810 04:47:36.755556 15284 mini_ranger.cc:161] Starting Ranger...
I20260810 04:47:36.770212 15284 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 04:47:37.195976 15284 mini_ranger.cc:87] Created ranger Postgres database
I20260810 04:47:37.196060 15284 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-admin
2026-08-10 04:47:37,451 [I] DB FLAVOR :POSTGRES
2026-08-10 04:47:37,452 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:47:37,452 [I] Checking connection..
2026-08-10 04:47:37,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 04:47:37,705 [I] Checking connection passed.
2026-08-10 04:47:37,705 [I] --------- Verifying version history table ---------
2026-08-10 04:47:37,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:47:37,961 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:47:37,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:47:38,193 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:47:38,193 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 04:47:38,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 04:47:38.439 UTC [3383] WARNING: there is no transaction in progress
2026-08-10 04:47:38,448 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 04:47:38,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:47:38,714 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 04:47:38,714 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:47:38,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 04:47:38,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 04:47:39,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8hhm.c.gcp-upstream.internal','N') ;"
2026-08-10 04:47:39,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:47:39,730 [I] Table x_portal_user does not exist in database ranger
2026-08-10 04:47:39,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:47:39,984 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 04:47:39,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:47:40,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:47:40,491 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 04:47:40,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 04:47:41.413 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.428 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.444 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.457 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.472 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.488 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.540 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.546 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.560 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.571 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.581 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.594 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.603 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.609 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.615 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:41.621 UTC [3563] WARNING: there is no transaction in progress
2026-08-10 04:47:42,217 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 04:47:42,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 04:47:42,468 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 04:47:42,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:47:42,729 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 04:47:42,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:47:42,988 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 04:47:42,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:47:43,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:47:43,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8hhm.c.gcp-upstream.internal';"
2026-08-10 04:47:43,725 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 04:47:43,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 04:47:43,979 [I] DB_PATCHES have already been applied
I20260810 04:47:43.985059 15284 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwUvI9j/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 04:47:43.985149 15284 mini_ranger.cc:193] Using host: 127.14.237.20
I20260810 04:47:43.988837 15284 mini_ranger.cc:227] Ranger admin URL: http://127.14.237.20:43395
Aug 10, 2026 4:47:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:47:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:47:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 4:47:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 4:47:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 4:47:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 4:47:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 04:47:45.361302 15284 mini_ranger.cc:160] Time spent starting Ranger: real 8.606s user 0.001s sys 0.009s
Aug 10, 2026 4:48:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 4:48:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 4:48:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 4:48:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 4:48:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 04:48:07.013902 15284 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.237.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.237.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/rangerkms_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.237.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/rangerkms_127.14.237.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.237.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.237.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.237.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/HTTP_127.14.237.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.237.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/HTTP_127.14.237.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 04:48:07 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337287, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 04:48:07.101903 15284 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 04:48:07.112121 15284 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 04:48:07.921222 15284 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 04:48:07.921330 15284 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms
I20260810 04:48:07.921347 15284 mini_ranger_kms.cc:227] Using postgres at 127.14.237.20:41147
2026-08-10 04:48:08,128 [I] DB FLAVOR :POSTGRES
2026-08-10 04:48:08,128 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:48:08,128 [I] Checking connection
2026-08-10 04:48:08,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 04:48:08,407 [I] connection success
2026-08-10 04:48:08,408 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 04:48:08,408 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 04:48:08,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 04:48:08,659 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 04:48:08,659 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:48:08,659 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 04:48:08,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.237.20:41147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 04:48:08,961 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 04:48:09.456552 15284 mini_ranger_kms.cc:310] Created kms service
I20260810 04:48:09.645851 15284 mini_ranger_kms.cc:326] Created kudu user
I20260810 04:48:09.695186 15284 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 04:48:09 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337287, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:10.092223 15284 mini_ranger_kms.cc:384] Added ranger policy
I20260810 04:48:10.092434 15284 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwUvI9j/build/release/bin/postgresql.jar:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwUvI9j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 04:48:10.092459 15284 mini_ranger_kms.cc:240] Using host: 127.14.237.20
I20260810 04:48:10.096725 15284 mini_ranger_kms.cc:276] Ranger KMS PID: 3868
I20260810 04:48:10.096807 15284 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.237.20:39357
Aug 10, 2026 4:48:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:48:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:48:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 4:48:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 4:48:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 4:48:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 04:48:12 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786337292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.237.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:12.843073 15284 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.741s user 0.007s sys 0.183s
I20260810 04:48:12.843174 15284 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 04:48:12.843219 15284 mini_ranger_kms.cc:401] 127.14.237.20:39357/kms/v1/keys
Aug 10 04:48:13 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.237.20@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
Aug 10 04:48:13 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.237.20@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
Aug 10 04:48:15 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:48:15.667670 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:40201
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:40201
--ranger_config_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:39357/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:15.750200 3951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:15.750480 3951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:15.750516 3951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:15.751645 3951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:15.751713 3951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:15.751744 3951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:15.751770 3951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:15.751830 3951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:15.753852 3951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.237.20:39357/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:40201
--ranger_config_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:15.754136 3951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:15.754374 3951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:15.760309 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:15.760318 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:15.760445 3951 server_base.cc:1034] running on GCE node
W20260810 04:48:15.760345 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:15.761049 3951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:15.761390 3951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:15.762555 3951 hybrid_clock.cc:648] HybridClock initialized: now 1786337295762544 us; error 33 us; skew 500 ppm
Aug 10 04:48:15 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337295, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:15.765496 3951 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:48:15.765969 3951 webserver.cc:460] Webserver started at http://127.14.237.62:33735/ using document root <none> and password file <none>
I20260810 04:48:15.766196 3951 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:15.766258 3951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:15.766414 3951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:48:15 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337295, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:15.818521 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18"
format_stamp: "Formatted at 2026-08-10 04:48:15 on dist-test-slave-8hhm"
server_key: "f37c12243506a95c5069ea18feb10955"
server_key_iv: "1cc2a2bc3483373c019ee565dfdcba03"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:15.819006 3951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18"
format_stamp: "Formatted at 2026-08-10 04:48:15 on dist-test-slave-8hhm"
server_key: "f37c12243506a95c5069ea18feb10955"
server_key_iv: "1cc2a2bc3483373c019ee565dfdcba03"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:15.820580 3951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:48:15.831496 3974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:15.831707 3951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:15.831818 3951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18"
format_stamp: "Formatted at 2026-08-10 04:48:15 on dist-test-slave-8hhm"
server_key: "f37c12243506a95c5069ea18feb10955"
server_key_iv: "1cc2a2bc3483373c019ee565dfdcba03"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:15.831947 3951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:15.863161 3951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:15.865391 3951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:15.865564 3951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:15.870597 3951 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:40201
I20260810 04:48:15.870626 4055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:40201 every 8 connection(s)
I20260810 04:48:15.871059 3951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:48:15.871934 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:15.874018 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 3951
I20260810 04:48:15.874135 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:48:15.874348 15284 external_mini_cluster.cc:1428] Setting key d956380e1f2c83767a43c032d49b237f
I20260810 04:48:15.875291 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Bootstrap starting.
I20260810 04:48:15.876055 4056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:15.876389 4056 log.cc:824] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:15.877561 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: No bootstrap required, opened a new log
I20260810 04:48:15.879853 4056 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } }
I20260810 04:48:15.880016 4056 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:15.880080 4056 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b8dde6a6d0454da8f6c5c1849c5c18, State: Initialized, Role: FOLLOWER
I20260810 04:48:15.880260 4056 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } }
I20260810 04:48:15.880358 4056 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:48:15.880414 4056 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:48:15.880473 4056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:15.881189 4056 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } }
I20260810 04:48:15.881381 4056 leader_election.cc:302] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8b8dde6a6d0454da8f6c5c1849c5c18; no voters:
I20260810 04:48:15.881575 4056 leader_election.cc:288] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:48:15.881603 4067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:15.881812 4067 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 1 LEADER]: Becoming Leader. State: Replica: e8b8dde6a6d0454da8f6c5c1849c5c18, State: Running, Role: LEADER
I20260810 04:48:15.881914 4056 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:15.881948 4067 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } }
I20260810 04:48:15.882368 4067 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } } }
I20260810 04:48:15.882534 4067 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: This master's current role is: LEADER
Aug 10 04:48:15 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:15.882773 4067 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8b8dde6a6d0454da8f6c5c1849c5c18. Latest consensus state: current_term: 1 leader_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } } }
I20260810 04:48:15.882886 4067 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:15.883436 4080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:48:15.883531 4056 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 04:48:15.884052 4080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:48:15.892479 4080 catalog_manager.cc:1347] Generated new cluster ID: 18003f7c689c427786ccb68fe1010318
I20260810 04:48:15.892491 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:15.874897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.14.237.62:40201)
0810 04:48:15.875077 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:15.875087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:15.875186 (+ 99us) server_negotiation.cc:407] Connection header received
0810 04:48:15.875737 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:15.875745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:15.875796 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:15.875951 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:15.876745 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:15.878216 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:15.879382 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:15.879998 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:15.880106 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:15.883270 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:15.883274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:15.883275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:15.883286 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:15.889110 (+ 5824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:15.889731 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:15.889734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:15.889734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:15.889891 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:15.890172 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:15.890173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:15.890173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:15.890827 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:15.891530 (+ 703us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:15.891615 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 04:48:15.892112 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260810 04:48:15.892652 4080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:48:15.902653 4080 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:48:15.903406 4080 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:48:15.916582 4080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Generated new TSK 0
I20260810 04:48:15.916749 4080 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:48:16.624432 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:39357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40201
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:16.739522 4126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:16.739763 4126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:16.739804 4126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:16.740976 4126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:16.741020 4126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:16.741045 4126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:48:16.742645 4126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.237.20:39357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:16.743013 4126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:16.743297 4126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:16.748608 4134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:16.748857 4136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:16.749053 4132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:16.749598 4126 server_base.cc:1034] running on GCE node
I20260810 04:48:16.749827 4126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:16.750105 4126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:16.751323 4126 hybrid_clock.cc:648] HybridClock initialized: now 1786337296751300 us; error 33 us; skew 500 ppm
Aug 10 04:48:16 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337296, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:16.754719 4126 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:48:16.755265 4126 webserver.cc:460] Webserver started at http://127.14.237.1:46215/ using document root <none> and password file <none>
I20260810 04:48:16.755537 4126 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:16.755594 4126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:16.755735 4126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:48:16 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337296, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:16.768932 4126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "853293c032bd4c36943e2aba3fa36171"
format_stamp: "Formatted at 2026-08-10 04:48:16 on dist-test-slave-8hhm"
server_key: "c27fbcda0805b4363829b401737f5149"
server_key_iv: "8b8f485337f3962e7e9e73291a24efd3"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:16.769552 4126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "853293c032bd4c36943e2aba3fa36171"
format_stamp: "Formatted at 2026-08-10 04:48:16 on dist-test-slave-8hhm"
server_key: "c27fbcda0805b4363829b401737f5149"
server_key_iv: "8b8f485337f3962e7e9e73291a24efd3"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:16.777514 4126 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20260810 04:48:16.791543 4147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:16.791816 4126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:16.791927 4126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "853293c032bd4c36943e2aba3fa36171"
format_stamp: "Formatted at 2026-08-10 04:48:16 on dist-test-slave-8hhm"
server_key: "c27fbcda0805b4363829b401737f5149"
server_key_iv: "8b8f485337f3962e7e9e73291a24efd3"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:16.792027 4126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:16.805929 4126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:16.807931 4126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:16.808074 4126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:16.808312 4126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:16.808663 4126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:16.808710 4126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:16.808739 4126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:16.808756 4126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:16.815985 4126 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:45867
I20260810 04:48:16.816429 4126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:48:16.816937 4343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:45867 every 8 connection(s)
I20260810 04:48:16.817407 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 4126
I20260810 04:48:16.817510 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:48:16.817709 15284 external_mini_cluster.cc:1428] Setting key e85596f0222f9e1c12039e2b59557b63
Aug 10 04:48:16 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337296, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:16.828233 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:16.817154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40201 (local address 127.14.237.1:53431)
0810 04:48:16.817570 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:16.817633 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:16.818554 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:16.818979 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:16.818984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:16.819242 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:16.819537 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:16.819544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:16.820496 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:16.820497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:16.821782 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:16.821786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:16.822284 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:16.822285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:16.822457 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:16.823023 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:16.823033 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:16.824720 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:16.826710 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:16.826716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:16.826719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:16.826979 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:16.827191 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:16.827192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:16.827192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:16.827310 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:16.827582 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:16.827588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:16.827752 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 04:48:16.827853 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 04:48:16.827994 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":103,"threads_started":1}
I20260810 04:48:16.828953 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:16.817526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:53431 (local address 127.14.237.62:40201)
0810 04:48:16.818770 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:16.818773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:16.818781 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:48:16.818804 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:16.818804 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:16.818817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:16.818879 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:16.819635 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:16.820389 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:16.821886 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:16.822193 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:16.822259 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:16.824830 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:16.824831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:16.824832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:16.824839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:16.826607 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:16.827059 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:16.827060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:16.827060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:16.827117 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:16.827375 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:16.827376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:16.827376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:16.827504 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:16.827558 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:16.828773 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0810 04:48:16.828858 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":93,"threads_started":1}
I20260810 04:48:16.829550 4344 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40201
I20260810 04:48:16.829639 4344 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:16.829813 4344 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:16.830421 3997 ts_manager.cc:194] Registered new tserver with Master: 853293c032bd4c36943e2aba3fa36171 (127.14.237.1:45867)
I20260810 04:48:16.831291 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:53431
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:48:16.863853 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:39357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40201
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:48:16 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786337296, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 04:48:16.965867 4354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:16.966104 4354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:16.966135 4354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:16.967730 4354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:16.967825 4354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:16.967908 4354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:48:16.970049 4354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.237.20:39357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:16.970485 4354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:16.970803 4354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:16.977478 4365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:16.977847 4364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:16.978055 4368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:16.978577 4354 server_base.cc:1034] running on GCE node
I20260810 04:48:16.978789 4354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:16.979024 4354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:16.980201 4354 hybrid_clock.cc:648] HybridClock initialized: now 1786337296980176 us; error 35 us; skew 500 ppm
Aug 10 04:48:16 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337296, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:16.983453 4354 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:48:16.983918 4354 webserver.cc:460] Webserver started at http://127.14.237.2:39077/ using document root <none> and password file <none>
I20260810 04:48:16.984169 4354 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:16.984223 4354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:16.984412 4354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:48:16 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337296, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:16.997567 4354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "876ccfc830584712b765ff059035a8e9"
format_stamp: "Formatted at 2026-08-10 04:48:16 on dist-test-slave-8hhm"
server_key: "66dbfe560086238f0d3591d1a53a9b97"
server_key_iv: "0d7183e782cc7baa6159a21114f87b3b"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:16.998090 4354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "876ccfc830584712b765ff059035a8e9"
format_stamp: "Formatted at 2026-08-10 04:48:16 on dist-test-slave-8hhm"
server_key: "66dbfe560086238f0d3591d1a53a9b97"
server_key_iv: "0d7183e782cc7baa6159a21114f87b3b"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:17.001654 4354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 04:48:17.019519 4387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.019749 4354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260810 04:48:17.019893 4354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "876ccfc830584712b765ff059035a8e9"
format_stamp: "Formatted at 2026-08-10 04:48:16 on dist-test-slave-8hhm"
server_key: "66dbfe560086238f0d3591d1a53a9b97"
server_key_iv: "0d7183e782cc7baa6159a21114f87b3b"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:17.020038 4354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:17.036212 4354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:17.038717 4354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:17.038911 4354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:17.039176 4354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:17.039597 4354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:17.039670 4354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.039728 4354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:17.039773 4354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.046669 4354 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:45635
I20260810 04:48:17.047070 4354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:48:17.050848 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 4354
I20260810 04:48:17.050971 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:48:17.051172 15284 external_mini_cluster.cc:1428] Setting key 4cf1d47c2aac09a5271fbbfb8f10b1bd
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337296, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:17.057999 4592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:45635 every 8 connection(s)
I20260810 04:48:17.060860 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:17.047935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:59861 (local address 127.14.237.62:40201)
0810 04:48:17.048002 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:17.048004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:17.048273 (+ 269us) server_negotiation.cc:407] Connection header received
0810 04:48:17.049335 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:17.049336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:17.049350 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:17.049381 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:17.050241 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.051026 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.054268 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.054590 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.054646 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:17.057377 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:17.057379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:17.057379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:17.057386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:17.059268 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:17.059774 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:17.059775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:17.059775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:17.059834 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:17.060101 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:17.060102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:17.060103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:17.060240 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:17.060296 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:17.060657 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 04:48:17.060750 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:48:17.061439 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:17.047902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40201 (local address 127.14.237.2:59861)
0810 04:48:17.048103 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:17.048155 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:17.049175 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:17.049579 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:17.049583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:17.049846 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:17.050118 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:17.050125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.052536 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.052537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:17.054133 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:17.054136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.054744 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.054745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:17.054882 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:17.055519 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:17.055530 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:17.057272 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:17.059370 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:17.059376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:17.059378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:17.059679 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:17.059899 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:17.059900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:17.059900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:17.060026 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:17.060332 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:17.060338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:17.060492 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 04:48:17.060995 (+ 503us) client_negotiation.cc:240] Negotiation successful
0810 04:48:17.061158 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260810 04:48:17.062140 4597 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40201
I20260810 04:48:17.062235 4597 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:17.062423 4597 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:17.062871 3997 ts_manager.cc:194] Registered new tserver with Master: 876ccfc830584712b765ff059035a8e9 (127.14.237.2:45635)
I20260810 04:48:17.063339 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:59861
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:48:17.092041 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:39357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40201
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:17.214370 4603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:17.214751 4603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:17.214841 4603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:17.216550 4603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:17.216790 4603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:17.216881 4603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:48:17.219286 4603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.237.20:39357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:17.219916 4603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:17.220268 4603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:17.227301 4614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:17.227497 4613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:17.227778 4617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:17.228719 4603 server_base.cc:1034] running on GCE node
I20260810 04:48:17.229010 4603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:17.229352 4603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:17.231504 4603 hybrid_clock.cc:648] HybridClock initialized: now 1786337297231484 us; error 44 us; skew 500 ppm
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:17.236848 4603 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:48:17.237382 4603 webserver.cc:460] Webserver started at http://127.14.237.3:37591/ using document root <none> and password file <none>
I20260810 04:48:17.237653 4603 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:17.237716 4603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:17.237848 4603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:17.253263 4603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "af6e79a94d01453e9e3d11c3703b3df9"
format_stamp: "Formatted at 2026-08-10 04:48:17 on dist-test-slave-8hhm"
server_key: "940ede9f568c399ff7f1f04d59c39e9c"
server_key_iv: "e527aface20ab8aec823c61415e4aefb"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:17.253753 4603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "af6e79a94d01453e9e3d11c3703b3df9"
format_stamp: "Formatted at 2026-08-10 04:48:17 on dist-test-slave-8hhm"
server_key: "940ede9f568c399ff7f1f04d59c39e9c"
server_key_iv: "e527aface20ab8aec823c61415e4aefb"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:17.261664 4603 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.000s
I20260810 04:48:17.276552 4630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.276743 4603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 04:48:17.276875 4603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "af6e79a94d01453e9e3d11c3703b3df9"
format_stamp: "Formatted at 2026-08-10 04:48:17 on dist-test-slave-8hhm"
server_key: "940ede9f568c399ff7f1f04d59c39e9c"
server_key_iv: "e527aface20ab8aec823c61415e4aefb"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:17.276976 4603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:17.294879 4603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:17.303990 4603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:17.304168 4603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:17.304463 4603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:17.304853 4603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:17.304906 4603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.304947 4603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:17.304971 4603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.311903 4603 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:38009
I20260810 04:48:17.312363 4603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:48:17.313553 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 4603
I20260810 04:48:17.313678 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:48:17.313890 15284 external_mini_cluster.cc:1428] Setting key be24f4b57ca613b5dddbda6773e9b4b6
I20260810 04:48:17.315398 4833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:38009 every 8 connection(s)
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:17.337641 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:17.316569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:38641 (local address 127.14.237.62:40201)
0810 04:48:17.316618 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:17.316620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:17.316933 (+ 313us) server_negotiation.cc:407] Connection header received
0810 04:48:17.317970 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:17.317971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:17.317983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:17.318014 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:17.318860 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.319643 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.321085 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.321407 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.321480 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:17.332207 (+ 10727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:17.332209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:17.332210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:17.332218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:17.334044 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:17.336458 (+ 2414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:17.336460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:17.336460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:17.336529 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:17.336832 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:17.336834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:17.336834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:17.336979 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:17.337023 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:17.337444 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 04:48:17.337538 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:48:17.338435 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:17.316321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40201 (local address 127.14.237.3:38641)
0810 04:48:17.316810 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:17.316854 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:17.317841 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:17.318201 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:17.318206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:17.318462 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:17.318748 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:17.318754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.319756 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.319757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:17.320965 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:17.320968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.329341 (+ 8373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.329342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:17.329508 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:17.330219 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:17.330230 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:17.332085 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:17.335453 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:17.335461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:17.335464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:17.335789 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:17.336608 (+ 819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:17.336609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:17.336609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:17.336753 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:17.337087 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:17.337093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:17.337282 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 04:48:17.338004 (+ 722us) client_negotiation.cc:240] Negotiation successful
0810 04:48:17.338189 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":88,"threads_started":1}
I20260810 04:48:17.339147 4836 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40201
I20260810 04:48:17.339242 4836 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:17.339428 4836 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:17.339849 3997 ts_manager.cc:194] Registered new tserver with Master: af6e79a94d01453e9e3d11c3703b3df9 (127.14.237.3:38009)
I20260810 04:48:17.340365 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:38641
I20260810 04:48:17.347031 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:48:17.347710 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 4126
I20260810 04:48:17.359400 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 4354
I20260810 04:48:17.390532 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 4603
I20260810 04:48:17.399817 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 3951
I20260810 04:48:17.410786 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:40201
--webserver_interface=127.14.237.62
--webserver_port=33735
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:40201
--ranger_config_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:39357/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.237.20:39357/kms with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:17.495787 4840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:17.496014 4840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:17.496052 4840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:17.497207 4840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:17.497298 4840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:17.497329 4840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:17.497360 4840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:17.497385 4840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:17.498968 4840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.237.20:39357/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:40201
--ranger_config_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=33735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:17.499241 4840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:17.499461 4840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:17.504420 4846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:17.504428 4849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:17.504698 4847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:17.504904 4840 server_base.cc:1034] running on GCE node
I20260810 04:48:17.505247 4840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:17.505637 4840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:17.506888 4840 hybrid_clock.cc:648] HybridClock initialized: now 1786337297506870 us; error 25 us; skew 500 ppm
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:17.510021 4840 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:48:17.510558 4840 webserver.cc:460] Webserver started at http://127.14.237.62:33735/ using document root <none> and password file <none>
I20260810 04:48:17.510846 4840 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:17.510934 4840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:17.583810 4840 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:17.584702 4860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.584944 4840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:17.585047 4840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18"
format_stamp: "Formatted at 2026-08-10 04:48:15 on dist-test-slave-8hhm"
server_key: "f37c12243506a95c5069ea18feb10955"
server_key_iv: "1cc2a2bc3483373c019ee565dfdcba03"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:17.585554 4840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:17.594043 4840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:17.595721 4840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:17.595919 4840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:17.600402 4840 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:40201
I20260810 04:48:17.600440 4960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:40201 every 8 connection(s)
I20260810 04:48:17.600893 4840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:48:17.602116 4961 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:48:17.602802 4961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Bootstrap starting.
I20260810 04:48:17.607302 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 4840
I20260810 04:48:17.607641 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:39357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:45867
--local_ip_for_outbound_sockets=127.14.237.1
--tserver_master_addrs=127.14.237.62:40201
--webserver_port=46215
--webserver_interface=127.14.237.1
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.237.20:39357/kms with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:48:17.611204 4961 log.cc:824] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:17.612870 4961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:48:17.613157 4961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Bootstrap complete.
I20260810 04:48:17.615578 4961 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } }
I20260810 04:48:17.615840 4961 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b8dde6a6d0454da8f6c5c1849c5c18, State: Initialized, Role: FOLLOWER
I20260810 04:48:17.616072 4961 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } }
I20260810 04:48:17.616186 4961 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:48:17.616236 4961 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:48:17.616287 4961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:48:17.617287 4961 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } }
I20260810 04:48:17.617412 4961 leader_election.cc:302] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8b8dde6a6d0454da8f6c5c1849c5c18; no voters:
I20260810 04:48:17.617594 4961 leader_election.cc:288] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:48:17.617673 4968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:48:17.617861 4961 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:17.618052 4968 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [term 2 LEADER]: Becoming Leader. State: Replica: e8b8dde6a6d0454da8f6c5c1849c5c18, State: Running, Role: LEADER
I20260810 04:48:17.618172 4968 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } }
I20260810 04:48:17.618503 4974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } } }
I20260810 04:48:17.618611 4974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:17.618502 4968 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8b8dde6a6d0454da8f6c5c1849c5c18. Latest consensus state: current_term: 2 leader_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b8dde6a6d0454da8f6c5c1849c5c18" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 40201 } } }
I20260810 04:48:17.618851 4968 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:17.618830 4985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:48:17.619271 4961 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 04:48:17.619560 4985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:48:17.622749 4985 catalog_manager.cc:1259] Loaded cluster ID: 18003f7c689c427786ccb68fe1010318
I20260810 04:48:17.622884 4985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:48:17.625455 4985 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:48:17.625705 4985 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Loaded TSK: 0
I20260810 04:48:17.625942 4985 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:48:17.720964 4965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:17.721184 4965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:17.721222 4965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:17.722466 4965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:17.722534 4965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:17.722577 4965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:48:17.724020 4965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.237.20:39357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:45867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=46215
--tserver_master_addrs=127.14.237.62:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:17.724277 4965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:17.724509 4965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:17.732826 5028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:17.733182 5029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:17.733453 5032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:17.733830 4965 server_base.cc:1034] running on GCE node
I20260810 04:48:17.734131 4965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:17.734437 4965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:17.735627 4965 hybrid_clock.cc:648] HybridClock initialized: now 1786337297735599 us; error 40 us; skew 500 ppm
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:17.738462 4965 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:48:17.738991 4965 webserver.cc:460] Webserver started at http://127.14.237.1:46215/ using document root <none> and password file <none>
I20260810 04:48:17.739332 4965 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:17.739429 4965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:17.757342 4965 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260810 04:48:17.758399 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.758545 4965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:48:17.758620 4965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "853293c032bd4c36943e2aba3fa36171"
format_stamp: "Formatted at 2026-08-10 04:48:16 on dist-test-slave-8hhm"
server_key: "c27fbcda0805b4363829b401737f5149"
server_key_iv: "8b8f485337f3962e7e9e73291a24efd3"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:17.759032 4965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:17.792024 4965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:17.794731 4965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:17.794912 4965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:17.795161 4965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:17.795596 4965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:17.795660 4965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.795698 4965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:17.795732 4965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.804090 4965 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:45867
I20260810 04:48:17.804339 5232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:45867 every 8 connection(s)
I20260810 04:48:17.804608 4965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:48:17.807806 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 4965
I20260810 04:48:17.808128 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:39357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:45635
--local_ip_for_outbound_sockets=127.14.237.2
--tserver_master_addrs=127.14.237.62:40201
--webserver_port=39077
--webserver_interface=127.14.237.2
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.237.20:39357/kms with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:17.818701 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:17.805489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40201 (local address 127.14.237.1:47577)
0810 04:48:17.805728 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:17.805778 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:17.806341 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:17.806868 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:17.806872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:17.807066 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:17.807273 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:17.807278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.808936 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.808938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:17.810228 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:17.810232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.811258 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.811259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:17.811395 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:17.812043 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:17.812054 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:17.813935 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:17.816451 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:17.816457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:17.816460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:17.816836 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:17.817158 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:17.817159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:17.817159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:17.817318 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:17.817921 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:17.817929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:17.818091 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 04:48:17.818216 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 04:48:17.818439 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":115,"threads_started":1}
I20260810 04:48:17.819607 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:17.805576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:47577 (local address 127.14.237.62:40201)
0810 04:48:17.805882 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:17.805891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:17.805927 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:48:17.806614 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:17.806627 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:17.806677 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:17.806822 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:17.807405 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.808803 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.810355 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:17.811004 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:17.811129 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:17.814075 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:17.814080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:17.814080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:17.814093 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:17.816331 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:17.816946 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:17.816949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:17.816950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:17.817068 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:17.817405 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:17.817406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:17.817406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:17.817747 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:17.817845 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:17.819218 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0810 04:48:17.819350 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260810 04:48:17.820560 5233 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40201
I20260810 04:48:17.820665 5233 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:17.820850 5233 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
W20260810 04:48:17.821812 5233 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:40201 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:48:17.821898 5233 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:17.821969 5233 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:17.822355 5233 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:17.822438 5233 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
W20260810 04:48:17.826857 5233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:48:17.917335 5238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:17.917625 5238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:17.917667 5238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:17.919291 5238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:17.919349 5238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:17.919387 5238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:48:17.921392 5238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.237.20:39357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:45635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=39077
--tserver_master_addrs=127.14.237.62:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:17.921712 5238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:17.921958 5238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:17.927321 5247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:17.927345 5244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:17.927850 5238 server_base.cc:1034] running on GCE node
W20260810 04:48:17.929667 5245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:17.929976 5238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:17.930263 5238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:17.937641 5238 hybrid_clock.cc:648] HybridClock initialized: now 1786337297937627 us; error 35 us; skew 500 ppm
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:17.940629 5238 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:48:17.941174 5238 webserver.cc:460] Webserver started at http://127.14.237.2:39077/ using document root <none> and password file <none>
I20260810 04:48:17.941504 5238 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:17.941608 5238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:48:17 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:17.960566 5238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:17.961727 5263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.961927 5238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:17.962025 5238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "876ccfc830584712b765ff059035a8e9"
format_stamp: "Formatted at 2026-08-10 04:48:16 on dist-test-slave-8hhm"
server_key: "66dbfe560086238f0d3591d1a53a9b97"
server_key_iv: "0d7183e782cc7baa6159a21114f87b3b"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:17.962491 5238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:17.987798 5238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:17.990201 5238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:17.990377 5238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:17.990643 5238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:17.991094 5238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:17.991183 5238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.991217 5238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:17.991237 5238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:17.999082 5238 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:45635
I20260810 04:48:17.999116 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:45635 every 8 connection(s)
I20260810 04:48:17.999646 5238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:48:18.000792 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 5238
I20260810 04:48:18.001077 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.237.20:39357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:38009
--local_ip_for_outbound_sockets=127.14.237.3
--tserver_master_addrs=127.14.237.62:40201
--webserver_port=37591
--webserver_interface=127.14.237.3
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.237.20:39357/kms with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:48:18 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337297, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:18.011766 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:18.000448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:60989 (local address 127.14.237.62:40201)
0810 04:48:18.000502 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:18.000504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:18.000755 (+ 251us) server_negotiation.cc:407] Connection header received
0810 04:48:18.001524 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:18.001526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:18.001542 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:18.001576 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:18.002454 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.003281 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.005328 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.005630 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.005684 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:18.008193 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:18.008194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:18.008195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:18.008203 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:18.010085 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:18.010604 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:18.010605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:18.010605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:18.010702 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:18.010999 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:18.011000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:18.011001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:18.011142 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:18.011187 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:18.011601 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 04:48:18.011689 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:48:18.012279 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:18.000397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40201 (local address 127.14.237.2:60989)
0810 04:48:18.000609 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:18.000658 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:18.001391 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:18.001799 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:18.001803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:18.002029 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:18.002324 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:18.002332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.003409 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.003410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:18.004692 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:18.004695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.005802 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.005802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:18.005905 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:18.006499 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:18.006509 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:18.008077 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:18.010208 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:18.010215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:18.010217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:18.010514 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:18.010788 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:18.010789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:18.010790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:18.010919 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:18.011239 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:18.011246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:18.011430 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 04:48:18.011945 (+ 515us) client_negotiation.cc:240] Negotiation successful
0810 04:48:18.012109 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260810 04:48:18.012827 5470 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40201
I20260810 04:48:18.012975 5470 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:18.013250 5470 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
W20260810 04:48:18.013945 5470 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:40201 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:48:18.014073 5470 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:18.014181 5470 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:18.014559 5470 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:18.014681 5470 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
W20260810 04:48:18.015005 5470 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:48:18.118486 5475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:18.118728 5475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:18.118760 5475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:18.120469 5475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:18.120528 5475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:18.120566 5475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:48:18.122728 5475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.237.20:39357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:38009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=37591
--tserver_master_addrs=127.14.237.62:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:18.122983 5475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:18.123243 5475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:18.129693 5486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:18.129694 5490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:18.129993 5485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:18.130894 5475 server_base.cc:1034] running on GCE node
I20260810 04:48:18.131127 5475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:18.131381 5475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:18.132568 5475 hybrid_clock.cc:648] HybridClock initialized: now 1786337298132540 us; error 40 us; skew 500 ppm
Aug 10 04:48:18 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337298, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:18.135928 5475 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:48:18.136502 5475 webserver.cc:460] Webserver started at http://127.14.237.3:37591/ using document root <none> and password file <none>
I20260810 04:48:18.136866 5475 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:18.136991 5475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:48:18 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337298, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:18.161437 5475 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 04:48:18.163458 5501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:18.163687 5475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260810 04:48:18.163820 5475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "af6e79a94d01453e9e3d11c3703b3df9"
format_stamp: "Formatted at 2026-08-10 04:48:17 on dist-test-slave-8hhm"
server_key: "940ede9f568c399ff7f1f04d59c39e9c"
server_key_iv: "e527aface20ab8aec823c61415e4aefb"
server_key_version: "kuduclusterkey@0"
I20260810 04:48:18.164326 5475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:18.186269 5475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:18.188690 5475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:18.188861 5475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:18.189101 5475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:18.189522 5475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:18.189581 5475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:18.189620 5475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:18.189647 5475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:18.196926 5475 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:38009
I20260810 04:48:18.197412 5475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:48:18.197384 5703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:38009 every 8 connection(s)
I20260810 04:48:18.199640 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 5475
Aug 10 04:48:18 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337298, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:18.208146 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:18.198269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:50513 (local address 127.14.237.62:40201)
0810 04:48:18.198339 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:18.198341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:18.198565 (+ 224us) server_negotiation.cc:407] Connection header received
0810 04:48:18.199099 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:18.199100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:18.199115 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:18.199178 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:18.199881 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.200467 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.201834 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.202205 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.202256 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:18.204621 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:18.204623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:18.204623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:18.204630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:18.206368 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:18.206919 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:18.206921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:18.206921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:18.206978 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:18.207277 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:18.207278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:18.207279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:18.207425 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:18.207472 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:18.207867 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 04:48:18.208008 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:48:18.208976 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:18.198269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:40201 (local address 127.14.237.3:50513)
0810 04:48:18.198444 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:18.198485 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:18.198988 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:18.199259 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:18.199263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:18.199431 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:18.199753 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:18.199761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.200639 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.200639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:18.201737 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:18.201740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.202382 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.202383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:18.202520 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:18.203089 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:18.203099 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:18.204512 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:18.206448 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:18.206455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:18.206458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:18.206820 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:18.207055 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:18.207056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:18.207056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:18.207191 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:18.207525 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:18.207532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:18.207696 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 04:48:18.208511 (+ 815us) client_negotiation.cc:240] Negotiation successful
0810 04:48:18.208686 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260810 04:48:18.210086 5704 heartbeater.cc:347] Connected to a master server at 127.14.237.62:40201
I20260810 04:48:18.210224 5704 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:18.210436 5704 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
W20260810 04:48:18.211133 5704 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:40201 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:48:18.211205 5704 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:18.211265 5704 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:18.211297 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:18.200592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.14.237.62:40201)
0810 04:48:18.201510 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:18.201512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:18.201519 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:48:18.201541 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:18.201542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:18.201553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:18.202400 (+ 847us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:18.202432 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.203519 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.205874 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:18.206198 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:18.206279 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:18.207793 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:18.207795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:18.207795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:18.207802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:18.209643 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:18.210057 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:18.210058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:18.210059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:18.210119 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:18.210364 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:18.210364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:18.210365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:18.210491 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:18.210648 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:18.211099 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 04:48:18.211185 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":68,"threads_started":1}
I20260810 04:48:18.211997 5704 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:18.212080 5704 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
W20260810 04:48:18.212406 5704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 04:48:18 dist-test-slave-8hhm krb5kdc[3214](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786337298, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:18.827100 5233 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:18.827287 5233 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:18.827690 4895 ts_manager.cc:194] Registered new tserver with Master: 853293c032bd4c36943e2aba3fa36171 (127.14.237.1:45867)
I20260810 04:48:18.828702 4895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:47577
I20260810 04:48:19.015219 5470 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:19.015417 5470 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:19.015750 4895 ts_manager.cc:194] Registered new tserver with Master: 876ccfc830584712b765ff059035a8e9 (127.14.237.2:45635)
I20260810 04:48:19.016192 4895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:60989
Aug 10 04:48:19 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337298, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
I20260810 04:48:19.212603 5704 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:19.212796 5704 heartbeater.cc:511] Master 127.14.237.62:40201 requested a full tablet report, sending...
I20260810 04:48:19.213264 4895 ts_manager.cc:194] Registered new tserver with Master: af6e79a94d01453e9e3d11c3703b3df9 (127.14.237.3:38009)
I20260810 04:48:19.213683 4895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:50513
I20260810 04:48:19.217670 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:48:19.226065 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.218716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.14.237.62:40201)
0810 04:48:19.218983 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:19.218984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:19.218990 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:48:19.219008 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:19.219009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:19.219019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:19.219106 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:19.219787 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.220283 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.221125 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.221439 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.221535 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.223033 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:19.223034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:19.223035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:19.223044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:19.224722 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:19.225144 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:19.225145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:19.225146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:19.225263 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:19.225475 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:19.225477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:19.225477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:19.225613 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:19.225743 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:19.225855 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 04:48:19.225946 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":64,"threads_started":1}
I20260810 04:48:19.228325 4895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:48:19.230293 4895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:48:19.238888 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.233529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.14.237.1:45867)
0810 04:48:19.234356 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:19.234359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:19.234368 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:48:19.234406 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:19.234407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:19.234421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:19.234525 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:19.234728 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.235933 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.237492 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.238259 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.238335 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.238400 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:19.238750 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 04:48:19.238791 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":74,"threads_started":1}
I20260810 04:48:19.239336 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.233304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:45867 (local address 127.0.0.1:38436)
0810 04:48:19.233967 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:19.234010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:19.234074 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:19.234531 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:19.234534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:19.234537 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:19.234625 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.234627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.236049 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.236050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.237387 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.237390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.238437 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.238438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.238523 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.238539 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 04:48:19.238624 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 04:48:19.238678 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":98,"threads_started":1}
I20260810 04:48:19.239854 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.233787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.14.237.2:45635)
0810 04:48:19.234313 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:19.234317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:19.234777 (+ 460us) server_negotiation.cc:407] Connection header received
0810 04:48:19.234817 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:19.234820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:19.234835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:19.234909 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:19.235153 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.236515 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.237779 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.238906 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.238982 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.239058 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:19.239729 (+ 671us) server_negotiation.cc:299] Negotiation successful
0810 04:48:19.239773 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":61,"threads_started":1}
I20260810 04:48:19.240089 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.233374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:45635 (local address 127.0.0.1:54918)
0810 04:48:19.234648 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:19.234659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:19.234711 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:19.235009 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:19.235010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:19.235012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:19.235066 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.235067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.236621 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.236622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.237682 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.237685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.239097 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.239098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.239158 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.239161 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:48:19.239962 (+ 801us) client_negotiation.cc:240] Negotiation successful
0810 04:48:19.240009 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"thread_start_us":68,"threads_started":1}
I20260810 04:48:19.240321 5124 tablet_service.cc:1467] Processing CreateTablet for tablet 11187c3674324495a24fbd00dc8fe10b (DEFAULT_TABLE table=test-table [id=8945326e52fa49e9bb7f3dfc9e8e9cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:19.240711 5124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11187c3674324495a24fbd00dc8fe10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:19.240898 5352 tablet_service.cc:1467] Processing CreateTablet for tablet 11187c3674324495a24fbd00dc8fe10b (DEFAULT_TABLE table=test-table [id=8945326e52fa49e9bb7f3dfc9e8e9cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:19.241521 5352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11187c3674324495a24fbd00dc8fe10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:19.245069 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.233588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:38009 (local address 127.0.0.1:32790)
0810 04:48:19.239318 (+ 5730us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:19.239329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:19.239375 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:19.240700 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:19.240701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:19.240703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:19.240753 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.240755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.242380 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.242381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.243437 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.243441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.244798 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.244799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.244874 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.244879 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:48:19.244932 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:48:19.244970 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5713,"thread_start_us":67,"threads_started":1}
I20260810 04:48:19.245440 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.233790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.14.237.3:38009)
0810 04:48:19.239524 (+ 5734us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:19.239529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:19.239539 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:48:19.239575 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:19.239576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:19.240510 (+ 934us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:19.240604 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:19.240888 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.242267 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.243550 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.244337 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.244421 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.245273 (+ 852us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:19.245307 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 04:48:19.245353 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5693,"thread_start_us":61,"threads_started":1}
I20260810 04:48:19.246129 5764 tablet_bootstrap.cc:492] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171: Bootstrap starting.
I20260810 04:48:19.246158 5591 tablet_service.cc:1467] Processing CreateTablet for tablet 11187c3674324495a24fbd00dc8fe10b (DEFAULT_TABLE table=test-table [id=8945326e52fa49e9bb7f3dfc9e8e9cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:19.246520 5591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11187c3674324495a24fbd00dc8fe10b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:19.247012 5764 tablet_bootstrap.cc:654] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:19.247403 5764 log.cc:824] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:19.248314 5764 tablet_bootstrap.cc:492] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171: No bootstrap required, opened a new log
I20260810 04:48:19.248387 5764 ts_tablet_manager.cc:1397] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:19.251071 5771 tablet_bootstrap.cc:492] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Bootstrap starting.
I20260810 04:48:19.251492 5764 raft_consensus.cc:348] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.251754 5771 tablet_bootstrap.cc:654] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:19.251863 5764 raft_consensus.cc:374] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:19.251978 5764 raft_consensus.cc:729] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853293c032bd4c36943e2aba3fa36171, State: Initialized, Role: FOLLOWER
I20260810 04:48:19.252141 5771 log.cc:824] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:19.252264 5764 consensus_queue.cc:260] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.252532 5766 tablet_bootstrap.cc:492] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Bootstrap starting.
I20260810 04:48:19.252722 5764 ts_tablet_manager.cc:1428] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260810 04:48:19.252900 5233 heartbeater.cc:503] Master 127.14.237.62:40201 was elected leader, sending a full tablet report...
I20260810 04:48:19.253269 5766 tablet_bootstrap.cc:654] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:19.254145 5771 tablet_bootstrap.cc:492] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: No bootstrap required, opened a new log
I20260810 04:48:19.254223 5771 ts_tablet_manager.cc:1397] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:48:19.256358 5771 raft_consensus.cc:348] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.256518 5771 raft_consensus.cc:374] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:19.256563 5771 raft_consensus.cc:729] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6e79a94d01453e9e3d11c3703b3df9, State: Initialized, Role: FOLLOWER
I20260810 04:48:19.256700 5771 consensus_queue.cc:260] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.257014 5704 heartbeater.cc:503] Master 127.14.237.62:40201 was elected leader, sending a full tablet report...
I20260810 04:48:19.257640 5766 log.cc:824] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:19.258037 5771 ts_tablet_manager.cc:1428] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 04:48:19.258781 5766 tablet_bootstrap.cc:492] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: No bootstrap required, opened a new log
I20260810 04:48:19.258925 5766 ts_tablet_manager.cc:1397] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 04:48:19.260803 5766 raft_consensus.cc:348] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.260958 5766 raft_consensus.cc:374] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:19.261005 5766 raft_consensus.cc:729] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 876ccfc830584712b765ff059035a8e9, State: Initialized, Role: FOLLOWER
I20260810 04:48:19.261142 5766 consensus_queue.cc:260] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.261504 5470 heartbeater.cc:503] Master 127.14.237.62:40201 was elected leader, sending a full tablet report...
I20260810 04:48:19.261821 5766 ts_tablet_manager.cc:1428] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260810 04:48:19.306116 5234 tablet.cc:2369] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 04:48:19 dist-test-slave-8hhm krb5kdc[3214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337298, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for HTTP/127.14.237.20@KRBTEST.COM
W20260810 04:48:19.448947 5705 tablet.cc:2369] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:48:19.452989 5773 raft_consensus.cc:482] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:19.453142 5773 raft_consensus.cc:504] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.453608 5773 leader_election.cc:288] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af6e79a94d01453e9e3d11c3703b3df9 (127.14.237.3:38009), 876ccfc830584712b765ff059035a8e9 (127.14.237.2:45635)
I20260810 04:48:19.457798 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.454220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:58483 (local address 127.14.237.2:45635)
0810 04:48:19.454276 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:19.454278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:19.454503 (+ 225us) server_negotiation.cc:407] Connection header received
0810 04:48:19.454613 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:19.454615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:19.454626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:19.454658 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:19.454910 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.455570 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.456843 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.457376 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.457452 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.457462 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:19.457676 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 04:48:19.457731 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:48:19.458065 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.454117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:45635 (local address 127.14.237.1:58483)
0810 04:48:19.454430 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:19.454443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:19.454536 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:19.454737 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:19.454738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:19.454740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:19.454822 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.454824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.455662 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.455663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.456734 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.456737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.457518 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.457520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.457592 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.457596 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:48:19.457931 (+ 335us) client_negotiation.cc:240] Negotiation successful
0810 04:48:19.457970 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":93,"thread_start_us":77,"threads_started":1}
I20260810 04:48:19.459256 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.453849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:38009 (local address 127.14.237.1:41745)
0810 04:48:19.454394 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:19.454406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:19.454495 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:19.454711 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:19.454712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:19.454714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:19.454800 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.454802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.455723 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.455724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.456935 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:19.456938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.459020 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.459021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:19.459091 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.459093 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:48:19.459136 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 04:48:19.459171 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":16,"thread_start_us":127,"threads_started":1}
I20260810 04:48:19.459198 5394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11187c3674324495a24fbd00dc8fe10b" candidate_uuid: "853293c032bd4c36943e2aba3fa36171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "876ccfc830584712b765ff059035a8e9" is_pre_election: true
I20260810 04:48:19.459270 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.454309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:41745 (local address 127.14.237.3:38009)
0810 04:48:19.454387 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:19.454389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:19.454464 (+ 75us) server_negotiation.cc:407] Connection header received
0810 04:48:19.454578 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:19.454579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:19.454589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:19.454633 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:19.454886 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.455600 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.457056 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.457606 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.457667 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.457675 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:19.459162 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0810 04:48:19.459210 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 04:48:19.459370 5394 raft_consensus.cc:2454] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853293c032bd4c36943e2aba3fa36171 in term 0.
I20260810 04:48:19.459612 5621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11187c3674324495a24fbd00dc8fe10b" candidate_uuid: "853293c032bd4c36943e2aba3fa36171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6e79a94d01453e9e3d11c3703b3df9" is_pre_election: true
I20260810 04:48:19.459781 5621 raft_consensus.cc:2454] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 853293c032bd4c36943e2aba3fa36171 in term 0.
I20260810 04:48:19.459762 5047 leader_election.cc:302] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853293c032bd4c36943e2aba3fa36171, 876ccfc830584712b765ff059035a8e9; no voters:
I20260810 04:48:19.459888 5773 raft_consensus.cc:2781] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:48:19.459967 5773 raft_consensus.cc:482] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:48:19.459990 5773 raft_consensus.cc:3037] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:19.460927 5773 raft_consensus.cc:504] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.461066 5773 leader_election.cc:288] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [CANDIDATE]: Term 1 election: Requested vote from peers af6e79a94d01453e9e3d11c3703b3df9 (127.14.237.3:38009), 876ccfc830584712b765ff059035a8e9 (127.14.237.2:45635)
I20260810 04:48:19.461302 5394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11187c3674324495a24fbd00dc8fe10b" candidate_uuid: "853293c032bd4c36943e2aba3fa36171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "876ccfc830584712b765ff059035a8e9"
I20260810 04:48:19.461395 5394 raft_consensus.cc:3037] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:19.462414 5394 raft_consensus.cc:2454] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853293c032bd4c36943e2aba3fa36171 in term 1.
I20260810 04:48:19.462641 5047 leader_election.cc:302] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 853293c032bd4c36943e2aba3fa36171, 876ccfc830584712b765ff059035a8e9; no voters:
I20260810 04:48:19.462728 5773 raft_consensus.cc:2781] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:19.462709 5621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11187c3674324495a24fbd00dc8fe10b" candidate_uuid: "853293c032bd4c36943e2aba3fa36171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6e79a94d01453e9e3d11c3703b3df9"
I20260810 04:48:19.462785 5621 raft_consensus.cc:3037] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:19.462875 5773 raft_consensus.cc:686] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [term 1 LEADER]: Becoming Leader. State: Replica: 853293c032bd4c36943e2aba3fa36171, State: Running, Role: LEADER
I20260810 04:48:19.462966 5773 consensus_queue.cc:237] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } }
I20260810 04:48:19.463686 5621 raft_consensus.cc:2454] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 853293c032bd4c36943e2aba3fa36171 in term 1.
I20260810 04:48:19.463872 4896 catalog_manager.cc:5495] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853293c032bd4c36943e2aba3fa36171 (127.14.237.1). New cstate: current_term: 1 leader_uuid: "853293c032bd4c36943e2aba3fa36171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "853293c032bd4c36943e2aba3fa36171" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 45867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 } health_report { overall_health: UNKNOWN } } }
W20260810 04:48:19.501020 5473 tablet.cc:2369] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:48:19.529470 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.525627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.14.237.1:45867)
0810 04:48:19.525673 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:19.525675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:19.525689 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:48:19.525787 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:19.525788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:19.525813 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:19.525901 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:48:19.526106 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.526902 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.528337 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:19.528663 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:19.528723 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:19.528929 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:48:19.529064 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:48:19.529149 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:19.529364 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 04:48:19.529386 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:48:19.542409 5394 raft_consensus.cc:1261] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 1 FOLLOWER]: Refusing update from remote peer 853293c032bd4c36943e2aba3fa36171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:19.542413 5621 raft_consensus.cc:1261] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 1 FOLLOWER]: Refusing update from remote peer 853293c032bd4c36943e2aba3fa36171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:19.542812 5773 consensus_queue.cc:1035] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6e79a94d01453e9e3d11c3703b3df9" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:19.542943 5785 consensus_queue.cc:1035] T 11187c3674324495a24fbd00dc8fe10b P 853293c032bd4c36943e2aba3fa36171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "876ccfc830584712b765ff059035a8e9" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:19.555352 5803 mvcc.cc:205] Tried to move back new op lower bound from 7316837578922426368 to 7316837578600919040. Current Snapshot: MvccSnapshot[applied={T|T < 7316837578922426368}]
I20260810 04:48:19.557324 5805 mvcc.cc:205] Tried to move back new op lower bound from 7316837578922426368 to 7316837578600919040. Current Snapshot: MvccSnapshot[applied={T|T < 7316837578922426368 or (T in {7316837578922426368})}]
I20260810 04:48:19.560249 4896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:48:19.560351 4896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:48:19.561276 4896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8b8dde6a6d0454da8f6c5c1849c5c18: Sending DeleteTablet for 3 replicas of tablet 11187c3674324495a24fbd00dc8fe10b
I20260810 04:48:19.561769 5591 tablet_service.cc:1514] Processing DeleteTablet for tablet 11187c3674324495a24fbd00dc8fe10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:19 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:32790
I20260810 04:48:19.562000 5352 tablet_service.cc:1514] Processing DeleteTablet for tablet 11187c3674324495a24fbd00dc8fe10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:19 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:54918
I20260810 04:48:19.562422 5124 tablet_service.cc:1514] Processing DeleteTablet for tablet 11187c3674324495a24fbd00dc8fe10b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:19 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:38436
I20260810 04:48:19.563128 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 4965
I20260810 04:48:19.573333 5814 tablet_replica.cc:330] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: stopping tablet replica
I20260810 04:48:19.573486 5814 raft_consensus.cc:2229] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:48:19.573629 5814 raft_consensus.cc:2258] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:19.573830 5816 tablet_replica.cc:330] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: stopping tablet replica
I20260810 04:48:19.573953 5816 raft_consensus.cc:2229] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:48:19.574069 5816 raft_consensus.cc:2258] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:19.580197 5814 ts_tablet_manager.cc:1905] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:19.582180 5814 ts_tablet_manager.cc:1918] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:19.582264 5814 log.cc:1196] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/11187c3674324495a24fbd00dc8fe10b
I20260810 04:48:19.582471 5816 ts_tablet_manager.cc:1905] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:19.582576 5814 ts_tablet_manager.cc:1939] T 11187c3674324495a24fbd00dc8fe10b P af6e79a94d01453e9e3d11c3703b3df9: Deleting consensus metadata
I20260810 04:48:19.583067 4864 catalog_manager.cc:4841] TS af6e79a94d01453e9e3d11c3703b3df9 (127.14.237.3:38009): tablet 11187c3674324495a24fbd00dc8fe10b (table test-table [id=8945326e52fa49e9bb7f3dfc9e8e9cf7]) successfully deleted
I20260810 04:48:19.584079 5816 ts_tablet_manager.cc:1918] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:19.584167 5816 log.cc:1196] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/11187c3674324495a24fbd00dc8fe10b
I20260810 04:48:19.584451 5816 ts_tablet_manager.cc:1939] T 11187c3674324495a24fbd00dc8fe10b P 876ccfc830584712b765ff059035a8e9: Deleting consensus metadata
I20260810 04:48:19.584878 4869 catalog_manager.cc:4841] TS 876ccfc830584712b765ff059035a8e9 (127.14.237.2:45635): tablet 11187c3674324495a24fbd00dc8fe10b (table test-table [id=8945326e52fa49e9bb7f3dfc9e8e9cf7]) successfully deleted
W20260810 04:48:19.585156 4871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:45867 (error 108)
I20260810 04:48:19.586126 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.585996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:45867 (local address 127.0.0.1:38456)
0810 04:48:19.586047 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:19.586081 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:48:19.586248 4871 catalog_manager.cc:4584] TS 853293c032bd4c36943e2aba3fa36171 (127.14.237.1:45867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11187c3674324495a24fbd00dc8fe10b: Network error: Client connection negotiation failed: client connection to 127.14.237.1:45867: connect: Connection refused (error 111)
I20260810 04:48:19.586465 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 5238
I20260810 04:48:19.600270 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 5475
I20260810 04:48:19.644244 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.644053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:45867 (local address 127.0.0.1:38464)
0810 04:48:19.644153 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:19.644192 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:48:19.647794 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:19.647687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:45867 (local address 127.0.0.1:38476)
0810 04:48:19.647732 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:19.647759 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:48:19.659907 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 4840
I20260810 04:48:19.675765 15284 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 04:48:20.023793 15284 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 04:48:20.023895 15284 mini_ranger.cc:66] Stopping Ranger...
I20260810 04:48:20.369619 15284 mini_ranger.cc:68] Stopped Ranger
2026-08-10 04:48:20.369 UTC [3264] LOG: received smart shutdown request
2026-08-10 04:48:20.374 UTC [3264] LOG: background worker "logical replication launcher" (PID 3277) exited with exit code 1
2026-08-10 04:48:20.375 UTC [3271] LOG: shutting down
2026-08-10 04:48:20.534 UTC [3264] LOG: database system is shut down
2026-08-10T04:48:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48830 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:48:20 dist-test-slave-8hhm krb5kdc[5831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:48:20 dist-test-slave-8hhm krb5kdc[5831](info): set up 2 sockets
Aug 10 04:48:20 dist-test-slave-8hhm krb5kdc[5831](info): commencing operation
krb5kdc: starting...
W20260810 04:48:21.932297 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:48:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:48:22.053786 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:43923
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:43923,127.14.237.61:42867,127.14.237.60:32847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:22.135560 5852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:22.135771 5852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:22.135794 5852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:22.136960 5852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:22.136999 5852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:22.137014 5852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:22.137081 5852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:22.137107 5852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:22.138645 5852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:43923,127.14.237.61:42867,127.14.237.60:32847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:43923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:22.138909 5852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:22.139144 5852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:22.144201 5862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:22.144225 5868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:22.144191 5863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:22.144877 5852 server_base.cc:1034] running on GCE node
I20260810 04:48:22.145179 5852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:22.145535 5852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:22.146684 5852 hybrid_clock.cc:648] HybridClock initialized: now 1786337302146671 us; error 39 us; skew 500 ppm
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:22.149163 5852 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:48:22.149682 5852 webserver.cc:460] Webserver started at http://127.14.237.62:45801/ using document root <none> and password file <none>
I20260810 04:48:22.149915 5852 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:22.149986 5852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:22.150151 5852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:22.151170 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "2c041612f4cf44a68838078ccaa26457"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "77b2ee502a4ec85b8a7c004fcd41adc8"
server_key_iv: "0497f27945c68a804898451ff8bdfb14"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.151543 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "2c041612f4cf44a68838078ccaa26457"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "77b2ee502a4ec85b8a7c004fcd41adc8"
server_key_iv: "0497f27945c68a804898451ff8bdfb14"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.152901 5852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 04:48:22.153776 5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:22.153952 5852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:48:22.154063 5852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "2c041612f4cf44a68838078ccaa26457"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "77b2ee502a4ec85b8a7c004fcd41adc8"
server_key_iv: "0497f27945c68a804898451ff8bdfb14"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.154142 5852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:22.173909 5852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:22.175879 5852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:22.176057 5852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:22.180472 5852 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:43923
I20260810 04:48:22.180510 5970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:43923 every 8 connection(s)
I20260810 04:48:22.180879 5852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:48:22.181875 5976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:22.183403 5976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:48:22.188833 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 5852
I20260810 04:48:22.188982 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:48:22.189160 15284 external_mini_cluster.cc:1428] Setting key 5d98c47a0064e271a0562a65e76b87e2
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:22.193403 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.184044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43923 (local address 127.0.0.1:36368)
0810 04:48:22.184269 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.184312 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.184807 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.185107 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.185110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.185420 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.185571 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.185576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.186629 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.186630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.187284 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.187286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.187779 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.187780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.187843 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.188412 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.188420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.190016 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.191871 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.191878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.191880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.192076 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.192315 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.192316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.192317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.192409 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.192750 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.192755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.192819 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.192926 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.193195 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":121,"thread_start_us":91,"threads_started":1}
I20260810 04:48:22.193414 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.184111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.14.237.62:43923)
0810 04:48:22.184280 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.184289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.184414 (+ 125us) server_negotiation.cc:407] Connection header received
0810 04:48:22.184970 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.184972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.184998 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.185100 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.185698 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.186504 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.187381 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.187682 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.187807 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.190127 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.190134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.190134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.190141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.191747 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.192160 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.192163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.192164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.192245 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.192476 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.192477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.192477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.192680 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.192769 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.192951 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.193149 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260810 04:48:22.194077 5976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:48:22.194458 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.194337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51628)
0810 04:48:22.194386 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.194429 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:48:22.194579 5887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
I20260810 04:48:22.195011 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.194918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51630)
0810 04:48:22.194965 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.194989 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:48:22.195117 5976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:42867: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.14.237.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab.
Entry for principal kudu/127.14.237.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab.
I20260810 04:48:22.215361 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.61:42867
--webserver_interface=127.14.237.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab
--principal=kudu/127.14.237.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:43923,127.14.237.61:42867,127.14.237.60:32847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:48:22.244338 5976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } attempt: 1
I20260810 04:48:22.244819 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.244682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51636)
0810 04:48:22.244751 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.244788 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 04:48:22.245198 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.245101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51640)
0810 04:48:22.245150 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.245174 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:48:22.245317 5976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:42867: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
W20260810 04:48:22.297917 5992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:22.298193 5992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:22.298240 5992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:22.299352 5992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:22.299428 5992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:22.299453 5992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:22.299474 5992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:22.299502 5992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:22.301093 5992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:43923,127.14.237.61:42867,127.14.237.60:32847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab
--principal=kudu/127.14.237.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.61:42867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.237.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:22.301414 5992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:22.301657 5992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:22.306491 6005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:22.306509 6004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:22.306491 6008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:22.306926 5992 server_base.cc:1034] running on GCE node
I20260810 04:48:22.307240 5992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:22.307540 5992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:22.308708 5992 hybrid_clock.cc:648] HybridClock initialized: now 1786337302308701 us; error 42 us; skew 500 ppm
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:22.311322 5992 init.cc:377] Logged in from keytab as kudu/127.14.237.61@KRBTEST.COM (short username kudu)
I20260810 04:48:22.311834 5992 webserver.cc:460] Webserver started at http://127.14.237.61:46479/ using document root <none> and password file <none>
I20260810 04:48:22.312065 5992 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:22.312136 5992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:22.312294 5992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:22.313444 5976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } attempt: 2
I20260810 04:48:22.313418 5992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/data/instance:
uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "5babefeeebbcfaa2c4794ee03f8b3ebd"
server_key_iv: "3ab0aa0f8de789c8119a79c4bd4b39ac"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.313838 5992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/wal/instance:
uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "5babefeeebbcfaa2c4794ee03f8b3ebd"
server_key_iv: "3ab0aa0f8de789c8119a79c4bd4b39ac"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.313855 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.313727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51642)
0810 04:48:22.313789 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.313827 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:48:22.314267 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.314170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51646)
0810 04:48:22.314219 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.314244 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:48:22.314360 5976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:42867: Network error: Client connection negotiation failed: client connection to 127.14.237.61:42867: connect: Connection refused (error 111)
I20260810 04:48:22.315326 5992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:48:22.316059 6021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:22.316251 5992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:48:22.316342 5992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/wal
uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "5babefeeebbcfaa2c4794ee03f8b3ebd"
server_key_iv: "3ab0aa0f8de789c8119a79c4bd4b39ac"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.316412 5992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:22.327044 5992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:22.328841 5992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:22.328999 5992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:22.333356 5992 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.61:42867
I20260810 04:48:22.333406 6121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.61:42867 every 8 connection(s)
I20260810 04:48:22.333763 5992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
I20260810 04:48:22.334640 6122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:22.336134 6122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:48:22.340292 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 5992
I20260810 04:48:22.340417 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-1/wal/instance
I20260810 04:48:22.340600 15284 external_mini_cluster.cc:1428] Setting key 7181c5c4c196d088ee5364ca15a11497
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:22.346067 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.336837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.14.237.62:43923)
0810 04:48:22.336915 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.336918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.337117 (+ 199us) server_negotiation.cc:407] Connection header received
0810 04:48:22.337570 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.337572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.337583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.337649 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.338229 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.338715 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.339642 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.339837 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.339892 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.342513 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.342514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.342515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.342523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.344453 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.344980 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.344982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.344983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.345045 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.345400 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.345401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.345401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.345525 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.345596 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.345885 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.345983 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:48:22.346241 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.336816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43923 (local address 127.0.0.1:36382)
0810 04:48:22.336995 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.337031 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.337490 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.337712 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.337715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.337957 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.338151 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.338155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.338796 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.338797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.339562 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.339563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.339913 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.339914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.340010 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.340631 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.340640 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.342393 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.344526 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.344552 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.344555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.344896 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.345141 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.345142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.345142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.345318 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.345601 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.345616 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.345750 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.345848 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.346016 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260810 04:48:22.346815 6122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.237.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.60@KRBTEST.COM" created.
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:48:22.355783 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.347154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.14.237.61:42867)
0810 04:48:22.347288 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.347292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.347308 (+ 16us) server_negotiation.cc:407] Connection header received
0810 04:48:22.347327 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.347329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.347347 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.347420 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.347686 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.348368 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.349038 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.349426 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.349506 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.352807 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.352815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.352816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.352829 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.354459 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.354920 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.354922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.354923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.354977 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.355185 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.355186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.355186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.355319 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.355381 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.355555 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.355696 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260810 04:48:22.355840 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.347070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51654)
0810 04:48:22.347117 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.347127 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.347168 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.347434 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.347435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.347568 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.347615 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.347616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.348467 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.348468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.348958 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.348959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.349502 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.349502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.349572 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.349962 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.349965 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.352692 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.354554 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.354555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.354556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.354834 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.355045 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.355046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.355046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.355122 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.355388 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.355390 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.355442 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.355520 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.355634 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 04:48:22.356463 6122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:48:22.356858 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.356711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50108)
0810 04:48:22.356769 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.356823 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 04:48:22.357030 6027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
I20260810 04:48:22.357534 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.357413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50118)
0810 04:48:22.357466 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.357501 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:48:22.357661 6122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:32847: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab.
Entry for principal kudu/127.14.237.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab.
I20260810 04:48:22.367733 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.60:32847
--webserver_interface=127.14.237.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab
--principal=kudu/127.14.237.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:43923,127.14.237.61:42867,127.14.237.60:32847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:48:22.405519 5976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } attempt: 3
I20260810 04:48:22.406899 6122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } attempt: 1
I20260810 04:48:22.407315 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.407172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50120)
0810 04:48:22.407231 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.407277 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 04:48:22.407629 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.407541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50132)
0810 04:48:22.407575 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.407606 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260810 04:48:22.407718 6122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:32847: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:48:22.413952 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.406267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.14.237.61:42867)
0810 04:48:22.406374 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.406377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.406398 (+ 21us) server_negotiation.cc:407] Connection header received
0810 04:48:22.406434 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.406435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.406454 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.406541 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.406934 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.407804 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.408579 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.408828 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.408921 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.411101 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.411103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.411103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.411111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.412611 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.412993 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.412994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.412995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.413061 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.413361 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.413362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.413363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.413484 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.413566 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.413744 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.413828 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 04:48:22.413954 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.406043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51664)
0810 04:48:22.406133 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.406148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.406220 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.406551 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.406552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.406750 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.406833 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.406836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.407898 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.407899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.408478 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.408480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.408919 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.408920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.409011 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.409508 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.409513 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.411005 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.412720 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.412721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.412722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.412913 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.413182 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.413183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.413184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.413284 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.413558 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.413560 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.413609 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.413701 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.413814 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 04:48:22.414400 5976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:48:22.414789 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.414685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50144)
0810 04:48:22.414737 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.414768 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:48:22.415132 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.415035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50158)
0810 04:48:22.415084 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.415110 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:48:22.415232 5976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:32847: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
I20260810 04:48:22.446369 5976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } attempt: 1
I20260810 04:48:22.446813 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.446684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50162)
0810 04:48:22.446746 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.446784 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 04:48:22.447163 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.447065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50164)
0810 04:48:22.447110 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.447138 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:48:22.447292 5976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:32847: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
W20260810 04:48:22.453960 6133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:22.454169 6133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:22.454191 6133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:22.455336 6133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:22.455377 6133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:22.455391 6133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:22.455404 6133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:22.455456 6133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:22.457050 6133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:43923,127.14.237.61:42867,127.14.237.60:32847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab
--principal=kudu/127.14.237.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.60:32847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.237.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:22.457340 6133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:22.457607 6133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:22.462565 6146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:22.462579 6143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:22.462579 6139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:22.463088 6133 server_base.cc:1034] running on GCE node
I20260810 04:48:22.463310 6133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:22.463546 6133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:22.464701 6133 hybrid_clock.cc:648] HybridClock initialized: now 1786337302464689 us; error 26 us; skew 500 ppm
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:22.467240 6133 init.cc:377] Logged in from keytab as kudu/127.14.237.60@KRBTEST.COM (short username kudu)
I20260810 04:48:22.467701 6133 webserver.cc:460] Webserver started at http://127.14.237.60:36361/ using document root <none> and password file <none>
I20260810 04:48:22.467906 6133 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:22.467960 6133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:22.468048 6133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:22.469012 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/data/instance:
uuid: "6f741d6860ea4752bb9d460fec46690e"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "00accc2a9ff79b32b7597a452d146b4b"
server_key_iv: "1ed445ec2aaa35e88e6d03e9950f2579"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.469395 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/wal/instance:
uuid: "6f741d6860ea4752bb9d460fec46690e"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "00accc2a9ff79b32b7597a452d146b4b"
server_key_iv: "1ed445ec2aaa35e88e6d03e9950f2579"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.470613 6133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:22.471247 6155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:22.471415 6133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:48:22.471483 6133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/wal
uuid: "6f741d6860ea4752bb9d460fec46690e"
format_stamp: "Formatted at 2026-08-10 04:48:22 on dist-test-slave-8hhm"
server_key: "00accc2a9ff79b32b7597a452d146b4b"
server_key_iv: "1ed445ec2aaa35e88e6d03e9950f2579"
server_key_version: "encryptionkey@0"
I20260810 04:48:22.471534 6133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:22.475847 6122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } attempt: 2
I20260810 04:48:22.476275 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.476132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50180)
0810 04:48:22.476197 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.476238 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 04:48:22.476645 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.476536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50192)
0810 04:48:22.476594 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.476622 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:48:22.476771 6122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:32847: Network error: Client connection negotiation failed: client connection to 127.14.237.60:32847: connect: Connection refused (error 111)
I20260810 04:48:22.486680 6133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:22.488955 6133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:22.489111 6133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:22.492985 6133 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.60:32847
I20260810 04:48:22.493088 6258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.60:32847 every 8 connection(s)
I20260810 04:48:22.493440 6133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
I20260810 04:48:22.494333 6259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:22.495872 6259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:22.503067 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 6133
I20260810 04:48:22.503194 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786337145367972-15284-0/minicluster-data/master-2/wal/instance
I20260810 04:48:22.503417 15284 external_mini_cluster.cc:1428] Setting key 2a86e600b5ddb1189d73506f073e4161
I20260810 04:48:22.504347 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.496557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.14.237.62:43923)
0810 04:48:22.496622 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.496624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.496780 (+ 156us) server_negotiation.cc:407] Connection header received
0810 04:48:22.497195 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.497196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.497206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.497321 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.497871 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.498341 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.499216 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.499389 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.499441 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.501536 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.501537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.501537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.501544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.502820 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.503317 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.503318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.503318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.503370 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.503670 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.503671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.503672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.503816 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.503888 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.504172 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.504260 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:48:22.504554 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.496504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43923 (local address 127.0.0.1:36388)
0810 04:48:22.496652 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.496691 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.497114 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.497373 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.497375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.497595 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.497776 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.497779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.498419 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.498420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.499138 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.499139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.499466 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.499467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.499571 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.500036 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.500044 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.501431 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.502912 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.502920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.502922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.503241 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.503441 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.503442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.503442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.503589 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.503882 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.503887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.504021 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.504143 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.504303 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260810 04:48:22.505122 6259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
I20260810 04:48:22.512034 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.505434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.14.237.61:42867)
0810 04:48:22.505476 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.505478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.505494 (+ 16us) server_negotiation.cc:407] Connection header received
0810 04:48:22.505574 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.505575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.505584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.505622 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.505916 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.506540 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.507283 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.507500 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.507570 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.509385 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.509386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.509386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.509405 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.510922 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.511336 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.511336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.511337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.511398 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.511608 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.511609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.511609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.511702 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.511750 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.511921 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.511975 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:48:22.512089 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.505374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:42867 (local address 127.0.0.1:51678)
0810 04:48:22.505450 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.505461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.505509 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.505643 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.505644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.505794 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.505848 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.505850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.506627 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.506628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.507205 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.507207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.507577 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.507578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.507620 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.508100 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.508105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.509305 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.511012 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.511013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.511014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.511248 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.511464 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.511465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.511465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.511527 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.511766 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.511767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.511824 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.511905 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.511992 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 04:48:22.512761 6259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:48:22.513121 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.503832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.14.237.60:32847)
0810 04:48:22.503970 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.503975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.504176 (+ 201us) server_negotiation.cc:407] Connection header received
0810 04:48:22.504195 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.504196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.504206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.504280 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.505037 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.506090 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.506952 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.507382 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.507430 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.510073 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.510081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.510082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.510088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.511886 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.512271 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.512274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.512275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.512338 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.512550 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.512551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.512551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.512687 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.512852 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.512923 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.513009 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:22.520730 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.513034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50206)
0810 04:48:22.513259 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.513268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.513339 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.513634 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.513635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.513740 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.513792 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.513793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.514552 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.514553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.515008 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.515010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.515413 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.515414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.515461 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.515862 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.515867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.517189 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.519547 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.519548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.519548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.519800 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.520031 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.520032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.520032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.520086 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.520346 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.520347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.520388 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.520467 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.520568 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260810 04:48:22.520808 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.513048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.14.237.60:32847)
0810 04:48:22.513192 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.513194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.513511 (+ 317us) server_negotiation.cc:407] Connection header received
0810 04:48:22.513529 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.513529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.513539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.513616 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.513883 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.514471 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.515107 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.515330 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.515412 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.517309 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.517309 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.517310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.517315 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.519469 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.519904 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.519905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.519906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.519960 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.520152 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.520153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.520154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.520276 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.520352 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.520506 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.520640 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":125,"thread_start_us":101,"threads_started":1}
I20260810 04:48:22.521055 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.514096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.14.237.62:43923)
0810 04:48:22.514169 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.514172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.514179 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:48:22.514201 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.514202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.514210 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.514259 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.514713 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.515228 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.515876 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.516082 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.516155 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.518328 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.518329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.518329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.518337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.519814 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.520229 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.520230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.520230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.520276 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.520480 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.520481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.520481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.520630 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.520825 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.520875 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.520949 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:48:22.522421 5976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } attempt: 2
I20260810 04:48:22.523166 6259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e: Bootstrap starting.
I20260810 04:48:22.523847 6259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:22.524175 6259 log.cc:824] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:22.525071 6259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e: No bootstrap required, opened a new log
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:48:22.527340 6259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.527489 6259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:22.527551 6259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f741d6860ea4752bb9d460fec46690e, State: Initialized, Role: FOLLOWER
I20260810 04:48:22.527683 6259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.527964 6272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.528028 6259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:22.528076 6272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:48:22.529847 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.522643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50220)
0810 04:48:22.522684 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.522694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.522727 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.523136 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.523137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.523290 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.523329 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.523330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.524206 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.524208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.524856 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.524859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.525361 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.525362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.525417 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.525822 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.525828 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.527109 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.528672 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.528673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.528674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.528886 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.529108 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.529109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.529109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.529182 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.529566 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.529567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.529616 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.529690 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.529778 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 04:48:22.530195 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.522993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.14.237.60:32847)
0810 04:48:22.523026 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.523028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.523035 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:48:22.523055 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.523055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.523065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.523129 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.523411 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.524125 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.524946 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.525284 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.525348 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.527211 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.527213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.527213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.527219 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.528588 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.528962 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.528963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.528963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.529017 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.529315 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.529316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.529317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.529486 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.529570 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.530051 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.530138 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:48:22.530966 6295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:48:22.531129 6295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:48:22.532177 5976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: Bootstrap starting.
I20260810 04:48:22.532930 5976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:22.532938 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.522983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.14.237.61:42867)
0810 04:48:22.523019 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.523020 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.523061 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:48:22.523084 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.523085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.523093 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.523137 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.523773 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.524393 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.525334 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.525597 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.525639 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.528823 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.528824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.528824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.528831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.530892 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.531260 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.531261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.531261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.531313 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.531491 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.531491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.531492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.531602 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.532326 (+ 724us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.532750 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.532829 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:48:22.533362 5976 log.cc:824] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:22.534293 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:48:22.534320 5976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: No bootstrap required, opened a new log
I20260810 04:48:22.533325 6133 master_runner.cc:186] Error getting master registration for 127.14.237.62:43923: OK, instance_id { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" instance_seqno: 1786337302179212 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:48:22.535496 6133 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:48:22.536191 5976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.536329 5976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:22.536381 5976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c041612f4cf44a68838078ccaa26457, State: Initialized, Role: FOLLOWER
I20260810 04:48:22.536485 5976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.536736 6303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.536784 5976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:22.536834 6303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:48:22.538607 6320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:48:22.538698 6320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:48:22.539451 5852 master_runner.cc:186] Error getting master registration for 127.14.237.61:42867: OK, instance_id { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" instance_seqno: 1786337302332139 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:48:22.541644 5852 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:48:22.548216 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.538362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.14.237.60:32847)
0810 04:48:22.538398 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.538400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.539312 (+ 912us) server_negotiation.cc:407] Connection header received
0810 04:48:22.539395 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.539396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.539408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.539435 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.540231 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.540891 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.542814 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.543086 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.543125 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.545246 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.545248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.545248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.545255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.546791 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.547375 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.547376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.547376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.547438 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.547644 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.547645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.547646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.547758 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.547832 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.548052 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.548138 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:48:22.551927 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.538556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.14.237.62:43923)
0810 04:48:22.538732 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.538734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.539250 (+ 516us) server_negotiation.cc:407] Connection header received
0810 04:48:22.539272 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.539273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.539282 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.539334 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.542471 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.543178 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.544369 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.544646 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.544683 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.548002 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.548004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.548004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.548011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.550056 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.551125 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.551127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.551127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.551178 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.551366 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.551367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.551367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.551476 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.551556 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.551751 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.551840 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260810 04:48:22.553346 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.538822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.14.237.61:42867)
0810 04:48:22.538856 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.538858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.541961 (+ 3103us) server_negotiation.cc:407] Connection header received
0810 04:48:22.542053 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.542055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.542064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.542091 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.543489 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.544180 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.546150 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.546440 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.546478 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.548835 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.548837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.548837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.548844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.550764 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.552475 (+ 1711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.552476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.552476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.552529 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.552739 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.552739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.552740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.552859 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.552938 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.553158 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.553251 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:48:22.567929 6122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } attempt: 3
Aug 10 04:48:22 dist-test-slave-8hhm krb5kdc[5831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337302, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
I20260810 04:48:22.575804 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.568203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:32847 (local address 127.0.0.1:50234)
0810 04:48:22.568257 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:22.568268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:22.568307 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:22.568440 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:22.568441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:22.568600 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:22.568643 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.568644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.569522 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.569523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.570326 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:22.570328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.570798 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.570799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:22.570857 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.571325 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:22.571330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:22.572836 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:22.574672 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.574673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.574674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.574914 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.575145 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:22.575146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:22.575146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:22.575209 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:22.575496 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:22.575497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:22.575541 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:48:22.575615 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 04:48:22.575707 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:48:22.575819 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:22.568293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.14.237.60:32847)
0810 04:48:22.568331 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:22.568333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:22.568340 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:48:22.568360 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:22.568361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:22.568370 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:22.568428 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:22.568721 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.569440 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.570434 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:22.570718 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:22.570787 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:22.572936 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:22.572937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:22.572937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:22.572944 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:22.574590 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.574993 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.574994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.574995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.575054 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:22.575279 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:22.575281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:22.575281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:22.575418 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:22.575491 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:22.575654 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 04:48:22.575730 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:48:22.577844 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86: Bootstrap starting.
I20260810 04:48:22.578563 6122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:22.578869 6122 log.cc:824] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:22.579839 6122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86: No bootstrap required, opened a new log
I20260810 04:48:22.581809 6122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.581944 6122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:22.581979 6122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908f4e0c99cd4ca3bdc3204edd1b7a86, State: Initialized, Role: FOLLOWER
I20260810 04:48:22.582176 6122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.582491 6122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:22.582449 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
W20260810 04:48:22.584389 6354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:48:22.584484 6354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:48:22.584610 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:48:22.586459 5992 master_runner.cc:263] No leader master found from master 908f4e0c99cd4ca3bdc3204edd1b7a86
I20260810 04:48:22.586506 5992 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:48:22.648406 6272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:22.648554 6272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.648785 6272 leader_election.cc:288] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c041612f4cf44a68838078ccaa26457 (127.14.237.62:43923), 908f4e0c99cd4ca3bdc3204edd1b7a86 (127.14.237.61:42867)
I20260810 04:48:22.649511 6073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f741d6860ea4752bb9d460fec46690e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" is_pre_election: true
I20260810 04:48:22.649729 6073 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f741d6860ea4752bb9d460fec46690e in term 0.
I20260810 04:48:22.650115 6162 leader_election.cc:302] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f741d6860ea4752bb9d460fec46690e, 908f4e0c99cd4ca3bdc3204edd1b7a86; no voters:
I20260810 04:48:22.650393 5936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f741d6860ea4752bb9d460fec46690e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c041612f4cf44a68838078ccaa26457" is_pre_election: true
I20260810 04:48:22.650532 5936 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f741d6860ea4752bb9d460fec46690e in term 0.
I20260810 04:48:22.650876 6272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:48:22.650941 6272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:48:22.650969 6272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:22.652110 6272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.652240 6272 leader_election.cc:288] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [CANDIDATE]: Term 1 election: Requested vote from peers 2c041612f4cf44a68838078ccaa26457 (127.14.237.62:43923), 908f4e0c99cd4ca3bdc3204edd1b7a86 (127.14.237.61:42867)
I20260810 04:48:22.652665 6073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f741d6860ea4752bb9d460fec46690e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86"
I20260810 04:48:22.652756 6073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:22.653800 6073 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f741d6860ea4752bb9d460fec46690e in term 1.
I20260810 04:48:22.654042 6162 leader_election.cc:302] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f741d6860ea4752bb9d460fec46690e, 908f4e0c99cd4ca3bdc3204edd1b7a86; no voters:
I20260810 04:48:22.654192 5936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f741d6860ea4752bb9d460fec46690e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c041612f4cf44a68838078ccaa26457"
I20260810 04:48:22.654278 5936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:22.655256 5936 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f741d6860ea4752bb9d460fec46690e in term 1.
I20260810 04:48:22.656631 6272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:22.656812 6272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [term 1 LEADER]: Becoming Leader. State: Replica: 6f741d6860ea4752bb9d460fec46690e, State: Running, Role: LEADER
I20260810 04:48:22.656901 6272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } }
I20260810 04:48:22.657516 6272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f741d6860ea4752bb9d460fec46690e. Latest consensus state: current_term: 1 leader_uuid: "6f741d6860ea4752bb9d460fec46690e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.657622 6272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:22.659720 6358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:48:22.659967 6358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:48:22.661667 6073 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [term 1 FOLLOWER]: Refusing update from remote peer 6f741d6860ea4752bb9d460fec46690e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:22.662084 6356 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [LEADER]: Connected to new peer: Peer: permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:22.663511 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f741d6860ea4752bb9d460fec46690e. Latest consensus state: current_term: 1 leader_uuid: "6f741d6860ea4752bb9d460fec46690e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.663623 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:48:22.664062 5936 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [term 1 FOLLOWER]: Refusing update from remote peer 6f741d6860ea4752bb9d460fec46690e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:22.664379 6356 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:22.665719 6303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f741d6860ea4752bb9d460fec46690e. Latest consensus state: current_term: 1 leader_uuid: "6f741d6860ea4752bb9d460fec46690e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.665815 6303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:48:22.667434 6356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f741d6860ea4752bb9d460fec46690e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.667553 6356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:22.667636 6361 mvcc.cc:205] Tried to move back new op lower bound from 7316837591699542016 to 7316837591683276800. Current Snapshot: MvccSnapshot[applied={T|T < 7316837591699542016}]
I20260810 04:48:22.668206 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f741d6860ea4752bb9d460fec46690e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.668294 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:48:22.668610 6364 mvcc.cc:205] Tried to move back new op lower bound from 7316837591699542016 to 7316837591683276800. Current Snapshot: MvccSnapshot[applied={T|T < 7316837591699542016}]
I20260810 04:48:22.669618 6358 catalog_manager.cc:1347] Generated new cluster ID: 3a9a4bee29f64ab9814157d77cb54aaa
I20260810 04:48:22.669685 6358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260810 04:48:22.676841 5893 tablet.cc:2369] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:48:22.678030 6272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f741d6860ea4752bb9d460fec46690e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.678128 6272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:22.678602 6303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f741d6860ea4752bb9d460fec46690e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c041612f4cf44a68838078ccaa26457" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43923 } } peers { permanent_uuid: "908f4e0c99cd4ca3bdc3204edd1b7a86" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 42867 } } peers { permanent_uuid: "6f741d6860ea4752bb9d460fec46690e" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 32847 } } }
I20260810 04:48:22.678714 6303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:48:22.683213 6358 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:48:22.684015 6358 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:48:22.689043 6358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f741d6860ea4752bb9d460fec46690e: Generated new TSK 0
I20260810 04:48:22.689280 6358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:48:23.539149 6320 catalog_manager.cc:1259] Loaded cluster ID: 3a9a4bee29f64ab9814157d77cb54aaa
I20260810 04:48:23.539253 6320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: loading cluster ID for follower catalog manager: success
I20260810 04:48:23.540043 6320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: acquiring CA information for follower catalog manager: success
I20260810 04:48:23.540385 6320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c041612f4cf44a68838078ccaa26457: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:48:23.584945 6354 catalog_manager.cc:1259] Loaded cluster ID: 3a9a4bee29f64ab9814157d77cb54aaa
I20260810 04:48:23.585026 6354 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86: loading cluster ID for follower catalog manager: success
I20260810 04:48:23.585870 6354 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86: acquiring CA information for follower catalog manager: success
I20260810 04:48:23.586194 6354 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 908f4e0c99cd4ca3bdc3204edd1b7a86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:48:23.739154 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 5852
I20260810 04:48:23.751173 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 5992
I20260810 04:48:23.763351 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 6133
2026-08-10T04:48:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3116 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:48:23 dist-test-slave-8hhm krb5kdc[6400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:48:23 dist-test-slave-8hhm krb5kdc[6400](info): set up 2 sockets
Aug 10 04:48:23 dist-test-slave-8hhm krb5kdc[6400](info): commencing operation
krb5kdc: starting...
W20260810 04:48:25.020279 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:48:25 dist-test-slave-8hhm krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:48:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:48:25.146507 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35569
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:25.229122 6419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:25.229342 6419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:25.229364 6419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:25.230412 6419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:25.230445 6419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:25.230458 6419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:25.230470 6419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:25.230482 6419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:25.231969 6419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:25.232371 6419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:25.232606 6419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:25.237525 6425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:25.237568 6427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:25.237613 6419 server_base.cc:1034] running on GCE node
W20260810 04:48:25.237568 6430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:25.238092 6419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:25.238440 6419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:25.239585 6419 hybrid_clock.cc:648] HybridClock initialized: now 1786337305239561 us; error 43 us; skew 500 ppm
Aug 10 04:48:25 dist-test-slave-8hhm krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337305, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:25.244971 6419 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:48:25.245514 6419 webserver.cc:460] Webserver started at http://127.14.237.62:38617/ using document root <none> and password file <none>
I20260810 04:48:25.245748 6419 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:25.245795 6419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:25.245954 6419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:25.247080 6419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "bd27aa316f274bfab469c392fd296813"
format_stamp: "Formatted at 2026-08-10 04:48:25 on dist-test-slave-8hhm"
server_key: "058942f37cac4ba087c8411a3b4f206d"
server_key_iv: "c39c1ba9f39e0ab295f2a2cb7f166a16"
server_key_version: "encryptionkey@0"
I20260810 04:48:25.247495 6419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "bd27aa316f274bfab469c392fd296813"
format_stamp: "Formatted at 2026-08-10 04:48:25 on dist-test-slave-8hhm"
server_key: "058942f37cac4ba087c8411a3b4f206d"
server_key_iv: "c39c1ba9f39e0ab295f2a2cb7f166a16"
server_key_version: "encryptionkey@0"
I20260810 04:48:25.248881 6419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:48:25.249698 6441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:25.249856 6419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:25.249962 6419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "bd27aa316f274bfab469c392fd296813"
format_stamp: "Formatted at 2026-08-10 04:48:25 on dist-test-slave-8hhm"
server_key: "058942f37cac4ba087c8411a3b4f206d"
server_key_iv: "c39c1ba9f39e0ab295f2a2cb7f166a16"
server_key_version: "encryptionkey@0"
I20260810 04:48:25.250044 6419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:25.258898 6419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:25.261101 6419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:25.261297 6419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:25.265450 6419 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:35569
I20260810 04:48:25.265496 6530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:35569 every 8 connection(s)
I20260810 04:48:25.265861 6419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:48:25.266832 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:25.269143 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813: Bootstrap starting.
I20260810 04:48:25.269837 6531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:25.270155 6531 log.cc:824] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:25.270870 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813: No bootstrap required, opened a new log
I20260810 04:48:25.272022 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 6419
I20260810 04:48:25.272161 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:48:25.272379 15284 external_mini_cluster.cc:1428] Setting key 2fa368d95686618aade26b3011650a47
I20260810 04:48:25.273353 6531 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd27aa316f274bfab469c392fd296813" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35569 } }
I20260810 04:48:25.273563 6531 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:25.273656 6531 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd27aa316f274bfab469c392fd296813, State: Initialized, Role: FOLLOWER
I20260810 04:48:25.273799 6531 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd27aa316f274bfab469c392fd296813" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35569 } }
I20260810 04:48:25.273907 6531 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:48:25.273949 6531 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:48:25.274024 6531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:25.274853 6531 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd27aa316f274bfab469c392fd296813" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35569 } }
I20260810 04:48:25.275014 6531 leader_election.cc:302] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd27aa316f274bfab469c392fd296813; no voters:
I20260810 04:48:25.275262 6531 leader_election.cc:288] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:48:25.275295 6543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:25.275417 6543 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [term 1 LEADER]: Becoming Leader. State: Replica: bd27aa316f274bfab469c392fd296813, State: Running, Role: LEADER
I20260810 04:48:25.275588 6531 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:25.275585 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd27aa316f274bfab469c392fd296813" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35569 } }
I20260810 04:48:25.276273 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd27aa316f274bfab469c392fd296813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd27aa316f274bfab469c392fd296813" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35569 } } }
I20260810 04:48:25.276290 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd27aa316f274bfab469c392fd296813. Latest consensus state: current_term: 1 leader_uuid: "bd27aa316f274bfab469c392fd296813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd27aa316f274bfab469c392fd296813" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35569 } } }
I20260810 04:48:25.276387 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:25.276412 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:25.276598 6560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:48:25.277298 6560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:48:25.279086 6560 catalog_manager.cc:1347] Generated new cluster ID: 15fc41adc0b44d04aa0bf79ab3f2c703
I20260810 04:48:25.279155 6560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:48:25 dist-test-slave-8hhm krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:25.283941 6560 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:48:25.284688 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:25.273019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.14.237.62:35569)
0810 04:48:25.273273 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:25.273283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:25.273319 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:48:25.273445 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:25.273453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:25.273512 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:25.273685 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:25.275009 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:25.276921 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:25.277930 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:25.278476 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:25.278579 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:25.281088 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:25.281091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:25.281092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:25.281103 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:25.282800 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:25.283182 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:25.283186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:25.283186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:25.283306 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:25.283530 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:25.283531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:25.283531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:25.283833 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:25.284378 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:25.284437 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 04:48:25.284560 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
I20260810 04:48:25.284829 6560 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:48:25.285857 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:48:25.295604 6560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd27aa316f274bfab469c392fd296813: Generated new TSK 0
I20260810 04:48:25.295805 6560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:48:25.340145 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 6419
2026-08-10T04:48:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1581 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:48:25 dist-test-slave-8hhm krb5kdc[6592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:48:25 dist-test-slave-8hhm krb5kdc[6592](info): set up 2 sockets
Aug 10 04:48:25 dist-test-slave-8hhm krb5kdc[6592](info): commencing operation
krb5kdc: starting...
W20260810 04:48:26.592338 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:48:26 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:48:26.716130 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35221
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:35221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:26.796351 6610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:26.796666 6610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:26.796706 6610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:26.797853 6610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:26.797935 6610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:26.797956 6610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:26.797976 6610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:26.798002 6610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:26.799417 6610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:35221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:26.799669 6610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:26.799881 6610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:26.804697 6621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:26.804708 6622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:26.804684 6624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:26.804730 6610 server_base.cc:1034] running on GCE node
I20260810 04:48:26.805274 6610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:26.805573 6610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:26.806712 6610 hybrid_clock.cc:648] HybridClock initialized: now 1786337306806693 us; error 36 us; skew 500 ppm
Aug 10 04:48:26 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:26.812377 6610 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:48:26.812840 6610 webserver.cc:460] Webserver started at http://127.14.237.62:38005/ using document root <none> and password file <none>
I20260810 04:48:26.813069 6610 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:26.813135 6610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:26.813309 6610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:26.814405 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "ef3c616bf972499888b80f39d3b13619"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "065ee9eebf3dd1de65fd395d03e5f2d9"
server_key_iv: "06a7c89ae6f62dc3e1b320d7dd5ebee0"
server_key_version: "encryptionkey@0"
I20260810 04:48:26.814782 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "ef3c616bf972499888b80f39d3b13619"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "065ee9eebf3dd1de65fd395d03e5f2d9"
server_key_iv: "06a7c89ae6f62dc3e1b320d7dd5ebee0"
server_key_version: "encryptionkey@0"
I20260810 04:48:26.816131 6610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:48:26.816866 6632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:26.817054 6610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:26.817162 6610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "ef3c616bf972499888b80f39d3b13619"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "065ee9eebf3dd1de65fd395d03e5f2d9"
server_key_iv: "06a7c89ae6f62dc3e1b320d7dd5ebee0"
server_key_version: "encryptionkey@0"
I20260810 04:48:26.817267 6610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:26.842794 6610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:26.844565 6610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:26.844725 6610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:26.848908 6610 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:35221
I20260810 04:48:26.848948 6727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:35221 every 8 connection(s)
I20260810 04:48:26.849366 6610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:48:26.850337 6728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:26.850911 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 6610
I20260810 04:48:26.851038 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:48:26.851274 15284 external_mini_cluster.cc:1428] Setting key 2c74c3c49517fbf44fd7137729cfd8f3
I20260810 04:48:26.853643 6728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Bootstrap starting.
I20260810 04:48:26.854328 6728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:26.854609 6728 log.cc:824] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:26.855324 6728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: No bootstrap required, opened a new log
I20260810 04:48:26.857414 6728 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:26.857584 6728 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:26.857617 6728 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3c616bf972499888b80f39d3b13619, State: Initialized, Role: FOLLOWER
Aug 10 04:48:26 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:26.857743 6728 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:26.857842 6728 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:48:26.857865 6728 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:48:26.857933 6728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:26.858470 6728 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:26.858574 6728 leader_election.cc:302] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef3c616bf972499888b80f39d3b13619; no voters:
I20260810 04:48:26.858768 6728 leader_election.cc:288] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:48:26.858829 6739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:26.859009 6739 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 1 LEADER]: Becoming Leader. State: Replica: ef3c616bf972499888b80f39d3b13619, State: Running, Role: LEADER
I20260810 04:48:26.859035 6728 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:26.859120 6739 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:26.859438 6739 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef3c616bf972499888b80f39d3b13619. Latest consensus state: current_term: 1 leader_uuid: "ef3c616bf972499888b80f39d3b13619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } } }
I20260810 04:48:26.859568 6739 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:26.859818 6756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:48:26.860590 6756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:48:26.860553 6741 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef3c616bf972499888b80f39d3b13619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } } }
I20260810 04:48:26.860751 6741 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:26.862509 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:26.851848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.14.237.62:35221)
0810 04:48:26.852016 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:26.852024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:26.852048 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:48:26.852443 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:26.852450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:26.852494 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:26.852609 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:26.853413 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:26.854409 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:26.855235 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:26.855678 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:26.855797 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:26.858229 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:26.858233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:26.858233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:26.858241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:26.860297 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:26.860750 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:26.860753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:26.860754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:26.860993 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:26.861246 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:26.861247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:26.861247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:26.861529 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:26.862102 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:26.862171 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 04:48:26.862316 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260810 04:48:26.865585 6756 catalog_manager.cc:1347] Generated new cluster ID: 81eba298d3ba4674bb29b079b8b5abcd
I20260810 04:48:26.865660 6756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:48:26.875322 6756 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:48:26.875818 6756 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:48:26.877758 6756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Generated new TSK 0
I20260810 04:48:26.877930 6756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:48:26.910449 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35221
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:26.990988 6774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:26.991202 6774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:26.991238 6774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:26.992383 6774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:26.992447 6774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:26.992491 6774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:48:26.993937 6774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:26.994199 6774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:26.994488 6774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:26.999580 6782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:26.999610 6779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:26.999594 6784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:26.999902 6774 server_base.cc:1034] running on GCE node
I20260810 04:48:27.000082 6774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:27.000260 6774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:27.001394 6774 hybrid_clock.cc:648] HybridClock initialized: now 1786337307001381 us; error 30 us; skew 500 ppm
Aug 10 04:48:26 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:27.003903 6774 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:48:27.004310 6774 webserver.cc:460] Webserver started at http://127.14.237.1:38553/ using document root <none> and password file <none>
I20260810 04:48:27.004547 6774 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:27.004591 6774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:27.004774 6774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:27.005875 6774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "b486ea208abf445380002d9408bcd5d2"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "2bcca5e21bc56510f8154e61c31658e7"
server_key_iv: "7698ad1b97e21a8204ea87714dca89fd"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.006227 6774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "b486ea208abf445380002d9408bcd5d2"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "2bcca5e21bc56510f8154e61c31658e7"
server_key_iv: "7698ad1b97e21a8204ea87714dca89fd"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.007627 6774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:48:27.008334 6794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.008507 6774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:48:27.008603 6774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "b486ea208abf445380002d9408bcd5d2"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "2bcca5e21bc56510f8154e61c31658e7"
server_key_iv: "7698ad1b97e21a8204ea87714dca89fd"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.008683 6774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:27.033465 6774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:27.035233 6774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:27.035403 6774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:27.035630 6774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:27.035948 6774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:27.036002 6774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.036036 6774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:27.036063 6774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.041584 6774 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:43007
I20260810 04:48:27.041633 6993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:43007 every 8 connection(s)
I20260810 04:48:27.041930 6774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:48:27.045786 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 6774
I20260810 04:48:27.046013 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:48:27.046203 15284 external_mini_cluster.cc:1428] Setting key 01e68fc831ef4f3ad23f644be93c72cd
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337306, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:27.051841 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.042722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:34791 (local address 127.14.237.62:35221)
0810 04:48:27.042788 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.042790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.042959 (+ 169us) server_negotiation.cc:407] Connection header received
0810 04:48:27.043431 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.043432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.043441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.043498 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:27.044368 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.044853 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.045975 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.046310 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.046392 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.048558 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:27.048559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:27.048560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:27.048567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:27.050367 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.050770 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.050771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.050772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.050841 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.051064 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.051065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.051066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.051223 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:27.051306 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.051555 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.051669 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:48:27.051860 6998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.042646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:34791)
0810 04:48:27.042829 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.042869 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.043348 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.043549 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.043552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.044098 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:27.044290 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.044294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.044933 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.044934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.045773 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.045775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.046408 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.046413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.046564 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.047095 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:27.047103 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:27.048450 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:27.050445 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.050451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.050452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.050693 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.050906 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.050906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.050907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.051001 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.051296 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:27.051301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:27.051403 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.051502 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.051677 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260810 04:48:27.052433 6995 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35221
I20260810 04:48:27.052537 6995 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:27.052709 6995 heartbeater.cc:511] Master 127.14.237.62:35221 requested a full tablet report, sending...
I20260810 04:48:27.053267 6662 ts_manager.cc:194] Registered new tserver with Master: b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007)
I20260810 04:48:27.053839 6662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:34791
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:48:27.072297 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35221
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:27.153477 7003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:27.153699 7003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:27.153730 7003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:27.154899 7003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:27.154964 7003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:27.155010 7003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:48:27.156502 7003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:27.156821 7003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:27.157061 7003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:27.162144 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.162108 7016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.162119 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:27.162359 7003 server_base.cc:1034] running on GCE node
I20260810 04:48:27.162563 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:27.162810 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:27.163966 7003 hybrid_clock.cc:648] HybridClock initialized: now 1786337307163952 us; error 25 us; skew 500 ppm
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:27.166513 7003 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:48:27.166944 7003 webserver.cc:460] Webserver started at http://127.14.237.2:33563/ using document root <none> and password file <none>
I20260810 04:48:27.167153 7003 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:27.167215 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:27.167346 7003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:27.168325 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "decc3a782b014ce98153df183d8d0055"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "7aa57745e71bc2abd56be2c57c7c3e8e"
server_key_iv: "85c22433f9daf927fbff8729886370c3"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.168678 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "decc3a782b014ce98153df183d8d0055"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "7aa57745e71bc2abd56be2c57c7c3e8e"
server_key_iv: "85c22433f9daf927fbff8729886370c3"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.170023 7003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 04:48:27.170696 7028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.170863 7003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:48:27.170955 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "decc3a782b014ce98153df183d8d0055"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "7aa57745e71bc2abd56be2c57c7c3e8e"
server_key_iv: "85c22433f9daf927fbff8729886370c3"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.171030 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:27.196246 7003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:27.198220 7003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:27.198397 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:27.198637 7003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:27.198946 7003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:27.199019 7003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.199055 7003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:27.199083 7003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.204931 7003 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:37969
I20260810 04:48:27.204968 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:37969 every 8 connection(s)
I20260810 04:48:27.205384 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:48:27.207536 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 7003
I20260810 04:48:27.207648 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:48:27.207861 15284 external_mini_cluster.cc:1428] Setting key 508f5d6fcd31e881ff41c8ef565614a4
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:27.215981 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.206133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:51693 (local address 127.14.237.62:35221)
0810 04:48:27.206181 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.206183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.206381 (+ 198us) server_negotiation.cc:407] Connection header received
0810 04:48:27.206843 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.206844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.206854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.206921 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:27.208164 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.208769 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.210266 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.210451 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.210529 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.212993 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:27.212995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:27.212995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:27.213002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:27.214474 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.214999 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.215000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.215000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.215052 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.215376 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.215377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.215377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.215475 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:27.215525 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.215853 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.215917 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:48:27.216250 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.206089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:51693)
0810 04:48:27.206243 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.206293 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.206762 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.206991 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.206994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.207727 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:27.208047 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.208053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.208869 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.208871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.210172 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.210175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.210541 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.210542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.210664 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.211167 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:27.211175 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:27.212906 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:27.214561 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.214569 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.214571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.214916 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.215122 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.215122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.215123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.215306 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.215540 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:27.215546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:27.215707 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.215835 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.216001 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260810 04:48:27.216843 7224 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35221
I20260810 04:48:27.216936 7224 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:27.217123 7224 heartbeater.cc:511] Master 127.14.237.62:35221 requested a full tablet report, sending...
I20260810 04:48:27.217537 6662 ts_manager.cc:194] Registered new tserver with Master: decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969)
I20260810 04:48:27.217913 6662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:51693
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:48:27.235642 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35221
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:27.317765 7230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:27.317966 7230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:27.317988 7230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:27.319027 7230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:27.319062 7230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:27.319087 7230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:48:27.320600 7230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:27.320858 7230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:27.321043 7230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:27.325927 7236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.325937 7237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.325937 7240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:27.326222 7230 server_base.cc:1034] running on GCE node
I20260810 04:48:27.326383 7230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:27.326558 7230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:27.327694 7230 hybrid_clock.cc:648] HybridClock initialized: now 1786337307327679 us; error 29 us; skew 500 ppm
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:27.330281 7230 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:48:27.330739 7230 webserver.cc:460] Webserver started at http://127.14.237.3:35401/ using document root <none> and password file <none>
I20260810 04:48:27.330930 7230 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:27.330971 7230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:27.331058 7230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:27.332005 7230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "ca1dd71c489e467384baa66ca23c77aa"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "59fa9dcc113a2aa0d3e1239da52aa402"
server_key_iv: "ec3158528a423432d60c5ef9992e2d52"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.332316 7230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "ca1dd71c489e467384baa66ca23c77aa"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "59fa9dcc113a2aa0d3e1239da52aa402"
server_key_iv: "ec3158528a423432d60c5ef9992e2d52"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.333705 7230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:48:27.334378 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.334555 7230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:48:27.334635 7230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "ca1dd71c489e467384baa66ca23c77aa"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "59fa9dcc113a2aa0d3e1239da52aa402"
server_key_iv: "ec3158528a423432d60c5ef9992e2d52"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.334687 7230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:27.343160 7230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:27.344801 7230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:27.344901 7230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:27.345053 7230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:27.345412 7230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:27.345458 7230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.345479 7230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:27.345491 7230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.351465 7230 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:32963
I20260810 04:48:27.351522 7460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:32963 every 8 connection(s)
I20260810 04:48:27.351823 7230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:27.360378 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 7230
I20260810 04:48:27.360502 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:48:27.360703 15284 external_mini_cluster.cc:1428] Setting key 73d0b7e63b10008af9cb09b78f008e28
I20260810 04:48:27.360980 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.352531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:56477 (local address 127.14.237.62:35221)
0810 04:48:27.352621 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.352624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.352771 (+ 147us) server_negotiation.cc:407] Connection header received
0810 04:48:27.353206 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.353207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.353254 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.353333 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:27.354198 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.354688 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.355746 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.355935 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.355993 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.358109 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:27.358111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:27.358111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:27.358119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:27.359525 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.359932 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.359933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.359933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.359991 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.360267 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.360268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.360268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.360415 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:27.360497 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.360807 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.360892 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:48:27.361145 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.352489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:56477)
0810 04:48:27.352654 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.352685 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.353128 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.353391 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.353395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.353947 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:27.354120 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.354125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.354770 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.354771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.355664 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.355667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.356012 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.356013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.356110 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.356602 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:27.356610 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:27.358010 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:27.359629 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.359634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.359635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.359848 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.360055 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.360055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.360055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.360194 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.360490 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:27.360497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:27.360665 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.360771 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.360923 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260810 04:48:27.361676 7461 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35221
I20260810 04:48:27.361778 7461 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:27.361943 7461 heartbeater.cc:511] Master 127.14.237.62:35221 requested a full tablet report, sending...
I20260810 04:48:27.362323 6661 ts_manager.cc:194] Registered new tserver with Master: ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963)
I20260810 04:48:27.362668 6661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:56477
I20260810 04:48:27.372596 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:48:27.379768 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.374001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.14.237.62:35221)
0810 04:48:27.374055 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.374057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.374138 (+ 81us) server_negotiation.cc:407] Connection header received
0810 04:48:27.374184 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.374185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.374194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.374254 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:27.374859 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.375327 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.376016 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.376209 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.376268 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.377415 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:27.377417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:27.377417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:27.377423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:27.378814 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.379123 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.379124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.379124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.379173 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.379351 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.379351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.379352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.379479 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:27.379552 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.379651 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.379709 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 04:48:27.381426 6661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:48:27.382892 6661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:48:27.389611 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.385633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:32963 (local address 127.0.0.1:37252)
0810 04:48:27.386211 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.386220 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.386271 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.386445 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.386446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.386447 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:27.386504 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.386506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.387632 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.387633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.388468 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.388471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.389367 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.389368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.389435 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.389447 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.389499 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.389537 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":65,"threads_started":1}
I20260810 04:48:27.389650 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.385748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.14.237.3:32963)
0810 04:48:27.385912 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.385916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.386287 (+ 371us) server_negotiation.cc:407] Connection header received
0810 04:48:27.386335 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.386337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.386353 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.386439 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:27.386562 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.387532 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.388563 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.389287 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.389368 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.389423 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.389525 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.389567 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260810 04:48:27.389865 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.385154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:37969 (local address 127.0.0.1:57588)
0810 04:48:27.385368 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.385415 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.385483 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.386006 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.386009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.386011 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:27.386094 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.386095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.387411 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.387411 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.388713 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.388716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.389643 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.389644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.389714 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.389718 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.389768 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.389804 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":42,"thread_start_us":140,"threads_started":1}
I20260810 04:48:27.390407 7337 tablet_service.cc:1467] Processing CreateTablet for tablet 45de8db24f764eb699c3b3589bbe0891 (DEFAULT_TABLE table=test-table [id=d4c94b10e8a2497f8b00d8f148423d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:27.390575 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.385326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.14.237.2:37969)
0810 04:48:27.385537 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.385541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.385552 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:48:27.385598 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.385600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.385633 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.385713 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:27.386173 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.387307 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.388803 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.389547 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.389614 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.389665 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.390456 (+ 791us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.390489 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
I20260810 04:48:27.390714 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.385571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.14.237.1:43007)
0810 04:48:27.385971 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.385976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.385988 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:48:27.386188 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.386191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.386208 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.386286 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:27.386529 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.387228 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.389705 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.390327 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.390386 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.390420 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.390592 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.390635 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":75,"threads_started":1}
I20260810 04:48:27.390810 7337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45de8db24f764eb699c3b3589bbe0891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:27.390971 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.385448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43007 (local address 127.0.0.1:56070)
0810 04:48:27.385794 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.385805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.386028 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.386367 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.386368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.386371 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:27.386446 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.386448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.387333 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.387334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.388713 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.388716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.390412 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.390413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.390490 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.390493 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.390558 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.390906 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":65,"thread_start_us":78,"threads_started":1}
I20260810 04:48:27.391191 7114 tablet_service.cc:1467] Processing CreateTablet for tablet 45de8db24f764eb699c3b3589bbe0891 (DEFAULT_TABLE table=test-table [id=d4c94b10e8a2497f8b00d8f148423d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:27.391510 7114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45de8db24f764eb699c3b3589bbe0891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:27.391731 6876 tablet_service.cc:1467] Processing CreateTablet for tablet 45de8db24f764eb699c3b3589bbe0891 (DEFAULT_TABLE table=test-table [id=d4c94b10e8a2497f8b00d8f148423d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:27.392083 6876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45de8db24f764eb699c3b3589bbe0891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:27.394065 7493 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: Bootstrap starting.
I20260810 04:48:27.394728 7493 tablet_bootstrap.cc:654] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:27.395368 7497 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Bootstrap starting.
I20260810 04:48:27.395406 7493 log.cc:824] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:27.395784 7494 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: Bootstrap starting.
I20260810 04:48:27.396040 7497 tablet_bootstrap.cc:654] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:27.396198 7493 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: No bootstrap required, opened a new log
I20260810 04:48:27.396292 7493 ts_tablet_manager.cc:1397] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:27.396596 7494 tablet_bootstrap.cc:654] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:27.396677 7497 log.cc:824] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:27.397207 7494 log.cc:824] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:27.397550 7497 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: No bootstrap required, opened a new log
I20260810 04:48:27.397634 7497 ts_tablet_manager.cc:1397] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:27.397987 7494 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: No bootstrap required, opened a new log
I20260810 04:48:27.398072 7494 ts_tablet_manager.cc:1397] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:27.398486 7493 raft_consensus.cc:348] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.398661 7493 raft_consensus.cc:374] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:27.398698 7493 raft_consensus.cc:729] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decc3a782b014ce98153df183d8d0055, State: Initialized, Role: FOLLOWER
I20260810 04:48:27.398857 7493 consensus_queue.cc:260] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.399161 7493 ts_tablet_manager.cc:1428] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:48:27.399298 7224 heartbeater.cc:503] Master 127.14.237.62:35221 was elected leader, sending a full tablet report...
I20260810 04:48:27.399655 7497 raft_consensus.cc:348] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.399799 7497 raft_consensus.cc:374] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:27.399842 7497 raft_consensus.cc:729] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b486ea208abf445380002d9408bcd5d2, State: Initialized, Role: FOLLOWER
I20260810 04:48:27.399945 7497 consensus_queue.cc:260] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.399905 7494 raft_consensus.cc:348] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.400079 7494 raft_consensus.cc:374] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:27.400133 7494 raft_consensus.cc:729] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1dd71c489e467384baa66ca23c77aa, State: Initialized, Role: FOLLOWER
I20260810 04:48:27.400247 7497 ts_tablet_manager.cc:1428] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:48:27.400313 6995 heartbeater.cc:503] Master 127.14.237.62:35221 was elected leader, sending a full tablet report...
I20260810 04:48:27.400274 7494 consensus_queue.cc:260] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.400617 7461 heartbeater.cc:503] Master 127.14.237.62:35221 was elected leader, sending a full tablet report...
I20260810 04:48:27.400624 7494 ts_tablet_manager.cc:1428] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:48:27.420629 7506 raft_consensus.cc:482] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:27.420728 7506 raft_consensus.cc:504] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.421113 7506 leader_election.cc:288] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969), b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007)
I20260810 04:48:27.423933 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.421508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:35839 (local address 127.14.237.1:43007)
0810 04:48:27.421554 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.421556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.421653 (+ 97us) server_negotiation.cc:407] Connection header received
0810 04:48:27.421685 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.421686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.421698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.421754 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:27.421887 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.422329 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.423209 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.423582 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.423664 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.423674 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.423822 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.423866 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:48:27.423942 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.421409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43007 (local address 127.14.237.3:35839)
0810 04:48:27.421573 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.421584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.421639 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.421752 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.421752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.421754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:27.421817 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.421819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.422409 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.422410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.423116 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.423117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.423671 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.423672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.423740 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.423743 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.423793 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.423822 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260810 04:48:27.423980 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.421372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:35613 (local address 127.14.237.2:37969)
0810 04:48:27.421434 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.421436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.421449 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:48:27.421481 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.421482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.421492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.421555 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:27.421732 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.422248 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.423252 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.423594 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.423680 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.423688 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.423871 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.423909 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:48:27.424208 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.421298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:37969 (local address 127.14.237.3:35613)
0810 04:48:27.421358 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.421369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.421409 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.421568 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.421569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.421570 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:27.421643 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.421645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.422324 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.422325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.423136 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.423138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.423693 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.423693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.423774 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.423794 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.423853 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.423887 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":42}
I20260810 04:48:27.424268 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45de8db24f764eb699c3b3589bbe0891" candidate_uuid: "ca1dd71c489e467384baa66ca23c77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b486ea208abf445380002d9408bcd5d2" is_pre_election: true
I20260810 04:48:27.424434 6916 raft_consensus.cc:2454] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1dd71c489e467384baa66ca23c77aa in term 0.
I20260810 04:48:27.424613 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45de8db24f764eb699c3b3589bbe0891" candidate_uuid: "ca1dd71c489e467384baa66ca23c77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decc3a782b014ce98153df183d8d0055" is_pre_election: true
I20260810 04:48:27.424674 7256 leader_election.cc:302] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b486ea208abf445380002d9408bcd5d2, ca1dd71c489e467384baa66ca23c77aa; no voters:
I20260810 04:48:27.424793 7145 raft_consensus.cc:2454] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1dd71c489e467384baa66ca23c77aa in term 0.
I20260810 04:48:27.424860 7506 raft_consensus.cc:2781] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:48:27.424921 7506 raft_consensus.cc:482] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:48:27.424953 7506 raft_consensus.cc:3037] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:27.425729 7506 raft_consensus.cc:504] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.425868 7506 leader_election.cc:288] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [CANDIDATE]: Term 1 election: Requested vote from peers decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969), b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007)
I20260810 04:48:27.426015 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45de8db24f764eb699c3b3589bbe0891" candidate_uuid: "ca1dd71c489e467384baa66ca23c77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decc3a782b014ce98153df183d8d0055"
I20260810 04:48:27.426072 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45de8db24f764eb699c3b3589bbe0891" candidate_uuid: "ca1dd71c489e467384baa66ca23c77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b486ea208abf445380002d9408bcd5d2"
I20260810 04:48:27.426086 7145 raft_consensus.cc:3037] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:27.426151 6916 raft_consensus.cc:3037] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:27.426826 6916 raft_consensus.cc:2454] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1dd71c489e467384baa66ca23c77aa in term 1.
I20260810 04:48:27.426854 7145 raft_consensus.cc:2454] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1dd71c489e467384baa66ca23c77aa in term 1.
I20260810 04:48:27.426980 7256 leader_election.cc:302] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b486ea208abf445380002d9408bcd5d2, ca1dd71c489e467384baa66ca23c77aa; no voters:
I20260810 04:48:27.427071 7506 raft_consensus.cc:2781] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:27.427240 7506 raft_consensus.cc:686] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 1 LEADER]: Becoming Leader. State: Replica: ca1dd71c489e467384baa66ca23c77aa, State: Running, Role: LEADER
I20260810 04:48:27.427333 7506 consensus_queue.cc:237] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.428259 6661 catalog_manager.cc:5495] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1dd71c489e467384baa66ca23c77aa (127.14.237.3). New cstate: current_term: 1 leader_uuid: "ca1dd71c489e467384baa66ca23c77aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } health_report { overall_health: HEALTHY } } }
I20260810 04:48:27.438311 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.436123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.14.237.3:32963)
0810 04:48:27.436185 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.436187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.436213 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:48:27.436239 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.436240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.436254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.436305 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:48:27.436424 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.436914 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.437665 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.437836 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.437886 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.438009 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:48:27.438097 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:48:27.438172 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.438203 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.438219 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:48:27.439896 6916 raft_consensus.cc:1261] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Refusing update from remote peer ca1dd71c489e467384baa66ca23c77aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:27.439896 7145 raft_consensus.cc:1261] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 1 FOLLOWER]: Refusing update from remote peer ca1dd71c489e467384baa66ca23c77aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:27.440286 7510 consensus_queue.cc:1035] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:27.440414 7506 consensus_queue.cc:1035] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:27.442710 7515 mvcc.cc:205] Tried to move back new op lower bound from 7316837611271458816 to 7316837611223560192. Current Snapshot: MvccSnapshot[applied={T|T < 7316837611271458816}]
I20260810 04:48:27.443354 7517 mvcc.cc:205] Tried to move back new op lower bound from 7316837611271458816 to 7316837611223560192. Current Snapshot: MvccSnapshot[applied={T|T < 7316837611271458816}]
I20260810 04:48:27.444996 6657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:48:27.445084 6657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:48:27.445864 6657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Sending DeleteTablet for 3 replicas of tablet 45de8db24f764eb699c3b3589bbe0891
I20260810 04:48:27.446208 7114 tablet_service.cc:1514] Processing DeleteTablet for tablet 45de8db24f764eb699c3b3589bbe0891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:27 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:57588
I20260810 04:48:27.446225 6876 tablet_service.cc:1514] Processing DeleteTablet for tablet 45de8db24f764eb699c3b3589bbe0891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:27 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:56070
I20260810 04:48:27.446249 7337 tablet_service.cc:1514] Processing DeleteTablet for tablet 45de8db24f764eb699c3b3589bbe0891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:27 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:37252
I20260810 04:48:27.446458 7536 tablet_replica.cc:330] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: stopping tablet replica
I20260810 04:48:27.446467 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 6774
I20260810 04:48:27.446560 7536 raft_consensus.cc:2229] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:48:27.446661 7536 raft_consensus.cc:2258] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:27.458161 7536 ts_tablet_manager.cc:1905] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:27.460623 7536 ts_tablet_manager.cc:1918] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:27.460711 7536 log.cc:1196] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/45de8db24f764eb699c3b3589bbe0891
I20260810 04:48:27.460824 7537 tablet_replica.cc:330] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: stopping tablet replica
I20260810 04:48:27.460985 7537 raft_consensus.cc:2229] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:48:27.460999 7536 ts_tablet_manager.cc:1939] T 45de8db24f764eb699c3b3589bbe0891 P ca1dd71c489e467384baa66ca23c77aa: Deleting consensus metadata
I20260810 04:48:27.461162 7537 raft_consensus.cc:2258] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:27.461704 7537 ts_tablet_manager.cc:1905] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:27.462606 6633 catalog_manager.cc:4841] TS ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963): tablet 45de8db24f764eb699c3b3589bbe0891 (table test-table [id=d4c94b10e8a2497f8b00d8f148423d69]) successfully deleted
I20260810 04:48:27.463232 7537 ts_tablet_manager.cc:1918] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:27.463307 7537 log.cc:1196] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/45de8db24f764eb699c3b3589bbe0891
I20260810 04:48:27.463575 7537 ts_tablet_manager.cc:1939] T 45de8db24f764eb699c3b3589bbe0891 P decc3a782b014ce98153df183d8d0055: Deleting consensus metadata
I20260810 04:48:27.464001 6633 catalog_manager.cc:4841] TS decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969): tablet 45de8db24f764eb699c3b3589bbe0891 (table test-table [id=d4c94b10e8a2497f8b00d8f148423d69]) successfully deleted
W20260810 04:48:27.469297 6633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:43007 (error 108)
I20260810 04:48:27.469460 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 7003
I20260810 04:48:27.471302 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.471164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43007 (local address 127.0.0.1:56074)
0810 04:48:27.471215 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.471254 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 04:48:27.471413 6633 catalog_manager.cc:4584] TS b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45de8db24f764eb699c3b3589bbe0891: Network error: Client connection negotiation failed: client connection to 127.14.237.1:43007: connect: Connection refused (error 111)
I20260810 04:48:27.483217 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 7230
I20260810 04:48:27.498061 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 6610
I20260810 04:48:27.507474 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35221
--webserver_interface=127.14.237.62
--webserver_port=38005
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:35221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:27.589087 7541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:27.589344 7541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:27.589370 7541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:27.590595 7541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:27.590636 7541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:27.590648 7541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:27.590692 7541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:27.590709 7541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:27.592177 7541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:35221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:27.592424 7541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:27.592648 7541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:27.605818 7554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.605850 7555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.605793 7557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:27.606083 7541 server_base.cc:1034] running on GCE node
I20260810 04:48:27.606379 7541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:27.606693 7541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:27.607878 7541 hybrid_clock.cc:648] HybridClock initialized: now 1786337307607860 us; error 43 us; skew 500 ppm
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:27.610553 7541 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:48:27.610975 7541 webserver.cc:460] Webserver started at http://127.14.237.62:38005/ using document root <none> and password file <none>
I20260810 04:48:27.611205 7541 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:27.611281 7541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:27.612896 7541 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:48:27.613646 7570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.613818 7541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.613924 7541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "ef3c616bf972499888b80f39d3b13619"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "065ee9eebf3dd1de65fd395d03e5f2d9"
server_key_iv: "06a7c89ae6f62dc3e1b320d7dd5ebee0"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.614355 7541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:27.628866 7541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:27.631112 7541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:27.631280 7541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:27.635442 7541 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:35221
I20260810 04:48:27.635488 7676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:35221 every 8 connection(s)
I20260810 04:48:27.635878 7541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:48:27.637548 7677 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:48:27.638213 7677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Bootstrap starting.
I20260810 04:48:27.642771 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 7541
I20260810 04:48:27.643074 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:43007
--local_ip_for_outbound_sockets=127.14.237.1
--tserver_master_addrs=127.14.237.62:35221
--webserver_port=38553
--webserver_interface=127.14.237.1
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:48:27.646790 7677 log.cc:824] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:27.649508 7677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:48:27.649840 7677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Bootstrap complete.
I20260810 04:48:27.653913 7677 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:27.654323 7677 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3c616bf972499888b80f39d3b13619, State: Initialized, Role: FOLLOWER
I20260810 04:48:27.654521 7677 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:27.654614 7677 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:48:27.654675 7677 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:48:27.654733 7677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:48:27.656265 7677 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:27.656405 7677 leader_election.cc:302] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef3c616bf972499888b80f39d3b13619; no voters:
I20260810 04:48:27.656566 7677 leader_election.cc:288] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:48:27.656668 7681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:48:27.656800 7677 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:27.656808 7681 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 2 LEADER]: Becoming Leader. State: Replica: ef3c616bf972499888b80f39d3b13619, State: Running, Role: LEADER
I20260810 04:48:27.657039 7681 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:27.657397 7682 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef3c616bf972499888b80f39d3b13619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } } }
I20260810 04:48:27.657397 7681 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef3c616bf972499888b80f39d3b13619. Latest consensus state: current_term: 2 leader_uuid: "ef3c616bf972499888b80f39d3b13619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } } }
I20260810 04:48:27.657498 7682 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:27.657501 7681 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:27.658128 7698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:48:27.659030 7698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:48:27.659133 7698 catalog_manager.cc:1259] Loaded cluster ID: 81eba298d3ba4674bb29b079b8b5abcd
I20260810 04:48:27.659157 7698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:48:27.659433 7698 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 04:48:27.664609 7698 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 04:48:27.667016 7698 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:48:27.667196 7698 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Loaded TSK: 0
I20260810 04:48:27.667408 7698 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:48:27.732060 7679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:27.732257 7679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:27.732280 7679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:27.733490 7679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:27.733530 7679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:27.733557 7679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:48:27.735004 7679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:43007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=38553
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:27.735235 7679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:27.735488 7679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:27.741422 7720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.741508 7726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.741577 7716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:27.741765 7679 server_base.cc:1034] running on GCE node
I20260810 04:48:27.742246 7679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:27.742522 7679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:27.743675 7679 hybrid_clock.cc:648] HybridClock initialized: now 1786337307743659 us; error 32 us; skew 500 ppm
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:27.746366 7679 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:48:27.746817 7679 webserver.cc:460] Webserver started at http://127.14.237.1:38553/ using document root <none> and password file <none>
I20260810 04:48:27.747040 7679 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:27.747119 7679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:27.748519 7679 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:27.749266 7738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.749450 7679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:27.749560 7679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "b486ea208abf445380002d9408bcd5d2"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "2bcca5e21bc56510f8154e61c31658e7"
server_key_iv: "7698ad1b97e21a8204ea87714dca89fd"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.749898 7679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:27.761644 7679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:27.763805 7679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:27.763970 7679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:27.764204 7679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:27.764698 7753 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:48:27.766601 7679 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:48:27.766667 7679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:48:27.766703 7679 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:48:27.767357 7679 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:48:27.767412 7679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:48:27.767481 7753 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Bootstrap starting.
I20260810 04:48:27.774821 7679 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:43007
I20260810 04:48:27.775184 7679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:48:27.777397 7919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:43007 every 8 connection(s)
I20260810 04:48:27.778841 7753 log.cc:824] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:27.779631 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 7679
I20260810 04:48:27.779911 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:37969
--local_ip_for_outbound_sockets=127.14.237.2
--tserver_master_addrs=127.14.237.62:35221
--webserver_port=33563
--webserver_interface=127.14.237.2
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:48:27.781343 7753 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:48:27.781616 7753 tablet_bootstrap.cc:492] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Bootstrap complete.
I20260810 04:48:27.782831 7753 ts_tablet_manager.cc:1397] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:27.784044 7753 raft_consensus.cc:348] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.784343 7753 raft_consensus.cc:729] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b486ea208abf445380002d9408bcd5d2, State: Initialized, Role: FOLLOWER
I20260810 04:48:27.784495 7753 consensus_queue.cc:260] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } } peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } }
I20260810 04:48:27.784750 7753 ts_tablet_manager.cc:1428] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 04:48:27.788081 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.776013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:60205 (local address 127.14.237.62:35221)
0810 04:48:27.777674 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.777681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.777709 (+ 28us) server_negotiation.cc:407] Connection header received
0810 04:48:27.777810 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.777815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.777854 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.778000 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:27.778631 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.780020 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.781547 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.782130 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.782280 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.784545 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:27.784548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:27.784549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:27.784578 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:27.786394 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.786831 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.786834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.786835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.786929 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.787188 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.787189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.787189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.787395 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:27.787484 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.787787 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.787907 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1611,"thread_start_us":84,"threads_started":1}
I20260810 04:48:27.788081 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.775910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:60205)
0810 04:48:27.776108 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.776156 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.776597 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.778039 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.778043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.778338 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:27.778509 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.778514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.780155 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.780156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.781424 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.781427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.782250 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.782251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.782405 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.783001 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:27.783009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:27.784420 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:27.786503 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.786510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.786512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.786739 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.787001 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.787002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.787003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.787114 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.787477 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:27.787484 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:27.787615 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.787719 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.787890 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260810 04:48:27.788796 7920 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35221
I20260810 04:48:27.788908 7920 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:27.789073 7920 heartbeater.cc:511] Master 127.14.237.62:35221 requested a full tablet report, sending...
I20260810 04:48:27.789815 7610 ts_manager.cc:194] Registered new tserver with Master: b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007)
I20260810 04:48:27.790875 7610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:60205
I20260810 04:48:27.791877 7920 heartbeater.cc:503] Master 127.14.237.62:35221 was elected leader, sending a full tablet report...
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:48:27.799041 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.790592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.14.237.1:43007)
0810 04:48:27.791398 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.791402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.791410 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:48:27.791440 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.791442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.791454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.791511 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:27.792085 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.792968 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.793842 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.794128 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.794214 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.796127 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:27.796135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:27.796136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:27.796143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:27.797801 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.798197 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.798200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.798200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.798262 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.798466 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.798467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.798467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.798596 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:27.798687 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.798878 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.798950 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":66,"threads_started":1}
I20260810 04:48:27.799041 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.790371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43007 (local address 127.0.0.1:56082)
0810 04:48:27.791187 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.791221 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.791280 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.791539 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.791543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.791911 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:27.791964 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.791967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.793065 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.793066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.793753 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.793756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.794214 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.794215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.794303 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.794757 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:27.794762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:27.796035 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:27.797906 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.797912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.797914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.798108 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.798332 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.798332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.798333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.798392 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.798676 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:27.798682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:27.798732 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.798827 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.798947 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":81,"threads_started":1}
I20260810 04:48:27.799770 7821 tablet_service.cc:1514] Processing DeleteTablet for tablet 45de8db24f764eb699c3b3589bbe0891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:27 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:56082
I20260810 04:48:27.800220 7942 tablet_replica.cc:330] stopping tablet replica
I20260810 04:48:27.800344 7942 raft_consensus.cc:2229] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:48:27.800411 7942 raft_consensus.cc:2258] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:27.800700 7942 ts_tablet_manager.cc:1905] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:27.802698 7942 ts_tablet_manager.cc:1918] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:27.802788 7942 log.cc:1196] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/45de8db24f764eb699c3b3589bbe0891
I20260810 04:48:27.802992 7942 ts_tablet_manager.cc:1939] T 45de8db24f764eb699c3b3589bbe0891 P b486ea208abf445380002d9408bcd5d2: Deleting consensus metadata
I20260810 04:48:27.803468 7571 catalog_manager.cc:4841] TS b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007): tablet 45de8db24f764eb699c3b3589bbe0891 (table test-table [id=d4c94b10e8a2497f8b00d8f148423d69]) successfully deleted
W20260810 04:48:27.873620 7929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:27.873901 7929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:27.873930 7929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:27.875128 7929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:27.875172 7929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:27.875226 7929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:48:27.876674 7929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:37969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=33563
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:27.876916 7929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:27.877156 7929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:27.882200 7948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.882212 7953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:27.882282 7951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:27.882489 7929 server_base.cc:1034] running on GCE node
I20260810 04:48:27.882695 7929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:27.882911 7929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:27.884049 7929 hybrid_clock.cc:648] HybridClock initialized: now 1786337307884043 us; error 36 us; skew 500 ppm
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:27.886552 7929 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:48:27.886929 7929 webserver.cc:460] Webserver started at http://127.14.237.2:33563/ using document root <none> and password file <none>
I20260810 04:48:27.887104 7929 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:27.887154 7929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:27.888401 7929 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:27.888924 7968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.889097 7929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:48:27.889207 7929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "decc3a782b014ce98153df183d8d0055"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "7aa57745e71bc2abd56be2c57c7c3e8e"
server_key_iv: "85c22433f9daf927fbff8729886370c3"
server_key_version: "encryptionkey@0"
I20260810 04:48:27.889573 7929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:27.901516 7929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:27.903539 7929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:27.903646 7929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:27.903831 7929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:27.904067 7929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:27.904098 7929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.904117 7929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:27.904130 7929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:27.909750 7929 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:37969
I20260810 04:48:27.909824 8162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:37969 every 8 connection(s)
I20260810 04:48:27.910207 7929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:48:27.912559 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 7929
I20260810 04:48:27.912859 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:32963
--local_ip_for_outbound_sockets=127.14.237.3
--tserver_master_addrs=127.14.237.62:35221
--webserver_port=35401
--webserver_interface=127.14.237.3
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:48:27 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337307, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:27.920302 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.910930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:56351 (local address 127.14.237.62:35221)
0810 04:48:27.910977 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:27.910980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:27.911161 (+ 181us) server_negotiation.cc:407] Connection header received
0810 04:48:27.911670 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:27.911672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:27.911687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:27.911744 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:27.912733 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.913379 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.914374 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.914590 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.914667 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.917078 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:27.917079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:27.917080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:27.917088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:27.918885 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.919275 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.919276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.919276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.919345 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:27.919582 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:27.919583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:27.919583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:27.919749 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:27.919829 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:27.920070 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 04:48:27.920171 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:48:27.920329 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:27.910885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:56351)
0810 04:48:27.911021 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:27.911070 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:27.911561 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:27.911822 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:27.911826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:27.912381 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:27.912638 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.912646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.913473 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.913474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.914271 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:27.914273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:27.914674 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:27.914675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:27.914847 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:27.915358 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:27.915369 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:27.916977 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:27.918974 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.918980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.918981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.919195 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.919418 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:27.919418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:27.919419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:27.919512 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:27.919821 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:27.919825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:27.919930 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 04:48:27.920009 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:48:27.920176 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260810 04:48:27.921072 8163 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35221
I20260810 04:48:27.921193 8163 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:27.921422 8163 heartbeater.cc:511] Master 127.14.237.62:35221 requested a full tablet report, sending...
I20260810 04:48:27.921856 7610 ts_manager.cc:194] Registered new tserver with Master: decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969)
I20260810 04:48:27.922223 7610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:56351
W20260810 04:48:27.996816 8166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:27.997002 8166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:27.997025 8166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:27.998150 8166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:27.998193 8166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:27.998246 8166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:48:27.999907 8166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:32963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=35401
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:28.000126 8166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:28.000357 8166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:28.005442 8176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:28.005429 8178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:28.005560 8166 server_base.cc:1034] running on GCE node
W20260810 04:48:28.005427 8172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:28.005883 8166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:28.006085 8166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:28.007221 8166 hybrid_clock.cc:648] HybridClock initialized: now 1786337308007203 us; error 41 us; skew 500 ppm
Aug 10 04:48:28 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:28.009799 8166 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:48:28.010245 8166 webserver.cc:460] Webserver started at http://127.14.237.3:35401/ using document root <none> and password file <none>
I20260810 04:48:28.010461 8166 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:28.010538 8166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:28.011843 8166 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:28.012413 8190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.012567 8166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.012660 8166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "ca1dd71c489e467384baa66ca23c77aa"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "59fa9dcc113a2aa0d3e1239da52aa402"
server_key_iv: "ec3158528a423432d60c5ef9992e2d52"
server_key_version: "encryptionkey@0"
I20260810 04:48:28.012959 8166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:28.029593 8166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:28.031890 8166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:28.031999 8166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:28.032167 8166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:28.032477 8166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:28.032512 8166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.032557 8166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:28.032584 8166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.038353 8166 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:32963
I20260810 04:48:28.038429 8380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:32963 every 8 connection(s)
I20260810 04:48:28.038748 8166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:48:28.039822 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 8166
Aug 10 04:48:28 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:28.047253 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.040217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.14.237.62:35221)
0810 04:48:28.040439 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:28.040441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:28.040447 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:48:28.040466 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:28.040467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:28.040476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:28.040550 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:28.041094 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.041776 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.042732 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.043038 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.043112 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.044392 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:28.044393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:28.044394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:28.044401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:28.045987 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:28.046351 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:28.046352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:28.046353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:28.046420 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:28.046648 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:28.046649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:28.046649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:28.046814 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:28.047000 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:28.047043 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:48:28.047141 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":57,"threads_started":1}
I20260810 04:48:28.048986 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.039468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:51027 (local address 127.14.237.62:35221)
0810 04:48:28.039548 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:28.039550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:28.039724 (+ 174us) server_negotiation.cc:407] Connection header received
0810 04:48:28.040304 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:28.040306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:28.040317 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:28.040380 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:28.041502 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.042259 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.043445 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.043654 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.043744 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.046190 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:28.046191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:28.046191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:28.046197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:28.047886 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:28.048315 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:28.048316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:28.048316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:28.048366 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:28.048547 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:28.048547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:28.048548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:28.048633 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:28.048679 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:28.048879 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 04:48:28.048933 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:48:28.049175 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.039427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:51027)
0810 04:48:28.039596 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.039641 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:28.040165 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:28.040461 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:28.040465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:28.041120 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:28.041409 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.041416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.042355 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.042356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.043356 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.043358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.043759 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.043760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.043877 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.044410 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:28.044418 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:28.046095 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:28.047953 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:28.047959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:28.047961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:28.048253 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:28.048416 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:28.048416 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:28.048417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:28.048500 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:28.048689 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:28.048693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:28.048793 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 04:48:28.048869 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:48:28.048975 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260810 04:48:28.049674 8382 heartbeater.cc:347] Connected to a master server at 127.14.237.62:35221
I20260810 04:48:28.049751 8382 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:28.049881 8382 heartbeater.cc:511] Master 127.14.237.62:35221 requested a full tablet report, sending...
I20260810 04:48:28.050184 7610 ts_manager.cc:194] Registered new tserver with Master: ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963)
I20260810 04:48:28.050551 7610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:51027
I20260810 04:48:28.059058 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:48:28.062008 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.059452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.14.237.62:35221)
0810 04:48:28.059486 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:28.059487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:28.059619 (+ 132us) server_negotiation.cc:407] Connection header received
0810 04:48:28.059697 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:28.059698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:28.059743 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:28.059828 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:48:28.059985 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.060448 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.061240 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.061432 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.061503 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.061678 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:48:28.061781 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:48:28.061895 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:28.061915 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 04:48:28.061931 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 04:48:28.062379 7610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:48:28.063958 7610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:48:28.066340 7821 tablet_service.cc:1467] Processing CreateTablet for tablet 87d10468e8e341c3828479bacd24063d (DEFAULT_TABLE table=test-table [id=ddc87b75aef44eb8a54f0f973e486981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:28.066594 7821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d10468e8e341c3828479bacd24063d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:28.068212 7753 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Bootstrap starting.
I20260810 04:48:28.068570 7753 tablet_bootstrap.cc:654] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:28.069159 7753 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: No bootstrap required, opened a new log
I20260810 04:48:28.069235 7753 ts_tablet_manager.cc:1397] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:28.069557 7753 raft_consensus.cc:348] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.069620 7753 raft_consensus.cc:374] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:28.069648 7753 raft_consensus.cc:729] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b486ea208abf445380002d9408bcd5d2, State: Initialized, Role: FOLLOWER
I20260810 04:48:28.069722 7753 consensus_queue.cc:260] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.069823 7753 ts_tablet_manager.cc:1428] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:48:28.070114 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.065789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:32963 (local address 127.0.0.1:37264)
0810 04:48:28.065848 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.065859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:28.065903 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:28.066700 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:28.066701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:28.066702 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:28.066758 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.066759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.067917 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.067919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.068954 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.068956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.069818 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.069819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.069889 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.069914 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 04:48:28.069965 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:48:28.070036 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:48:28.070485 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.066012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.14.237.3:32963)
0810 04:48:28.066526 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:28.066549 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:28.066558 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:48:28.066593 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:28.066595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:28.066610 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:28.066695 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:28.066830 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.067804 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.069062 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.069716 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.070257 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.070329 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:28.070359 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 04:48:28.070401 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":95,"threads_started":1}
I20260810 04:48:28.071206 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.065882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:37969 (local address 127.0.0.1:57596)
0810 04:48:28.066032 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.066043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:28.066091 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:28.066473 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:28.066475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:28.066477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:28.066527 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.066557 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.067664 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.067665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.068862 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.068866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.071009 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.071010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.071070 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.071074 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:48:28.071125 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:48:28.071155 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260810 04:48:28.071271 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.066011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.14.237.2:37969)
0810 04:48:28.066216 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:28.066221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:28.066233 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:48:28.066272 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:28.066274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:28.066291 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:28.066382 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:28.066648 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.067537 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.068962 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.070929 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.071000 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.071052 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:28.071147 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 04:48:28.071188 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260810 04:48:28.071281 8275 tablet_service.cc:1467] Processing CreateTablet for tablet 87d10468e8e341c3828479bacd24063d (DEFAULT_TABLE table=test-table [id=ddc87b75aef44eb8a54f0f973e486981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:28.071606 8275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d10468e8e341c3828479bacd24063d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:28.074517 8403 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: Bootstrap starting.
I20260810 04:48:28.075016 8053 tablet_service.cc:1467] Processing CreateTablet for tablet 87d10468e8e341c3828479bacd24063d (DEFAULT_TABLE table=test-table [id=ddc87b75aef44eb8a54f0f973e486981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:28.075160 8403 tablet_bootstrap.cc:654] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:28.075336 8053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d10468e8e341c3828479bacd24063d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:28.075768 8403 log.cc:824] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:28.076587 8403 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: No bootstrap required, opened a new log
I20260810 04:48:28.076681 8403 ts_tablet_manager.cc:1397] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:28.078233 8412 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Bootstrap starting.
I20260810 04:48:28.078233 8403 raft_consensus.cc:348] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.078413 8403 raft_consensus.cc:374] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:28.078455 8403 raft_consensus.cc:729] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1dd71c489e467384baa66ca23c77aa, State: Initialized, Role: FOLLOWER
I20260810 04:48:28.078596 8403 consensus_queue.cc:260] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.078905 8382 heartbeater.cc:503] Master 127.14.237.62:35221 was elected leader, sending a full tablet report...
I20260810 04:48:28.079037 8412 tablet_bootstrap.cc:654] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:28.079214 8403 ts_tablet_manager.cc:1428] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:48:28.079670 8412 log.cc:824] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:28.080473 8412 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: No bootstrap required, opened a new log
I20260810 04:48:28.080534 8412 ts_tablet_manager.cc:1397] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:28.081956 8412 raft_consensus.cc:348] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.082090 8412 raft_consensus.cc:374] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:28.082115 8412 raft_consensus.cc:729] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decc3a782b014ce98153df183d8d0055, State: Initialized, Role: FOLLOWER
I20260810 04:48:28.082204 8412 consensus_queue.cc:260] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.082458 8412 ts_tablet_manager.cc:1428] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:28.082484 8163 heartbeater.cc:503] Master 127.14.237.62:35221 was elected leader, sending a full tablet report...
W20260810 04:48:28.160760 8164 tablet.cc:2369] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:48:28.219226 8416 raft_consensus.cc:482] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:28.219383 8416 raft_consensus.cc:504] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.219772 8416 leader_election.cc:288] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969), b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007)
I20260810 04:48:28.223713 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.219983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:37969 (local address 127.14.237.3:33549)
0810 04:48:28.220052 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.220065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:28.220120 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:28.220307 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:28.220308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:28.220310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:28.220387 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.220389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.221351 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.221352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.222513 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.222516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.223357 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.223358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.223464 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.223468 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:48:28.223540 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:48:28.223595 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":15}
I20260810 04:48:28.223726 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.220080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:33549 (local address 127.14.237.2:37969)
0810 04:48:28.220145 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:28.220147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:28.220185 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:48:28.220216 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:28.220217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:28.220227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:28.220281 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:28.220468 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.221271 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.222680 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.223243 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.223323 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.223333 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:28.223600 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 04:48:28.223652 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:48:28.223731 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.220243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:50973 (local address 127.14.237.1:43007)
0810 04:48:28.220288 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:28.220290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:28.220354 (+ 64us) server_negotiation.cc:407] Connection header received
0810 04:48:28.220454 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:28.220455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:28.220484 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:28.220551 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:28.220704 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.221433 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.222680 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.223354 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.223459 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.223587 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:28.223615 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 04:48:28.223667 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:48:28.223724 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.220119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43007 (local address 127.14.237.3:50973)
0810 04:48:28.220295 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.220305 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:28.220360 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:28.220567 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:28.220567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:28.220569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:28.220623 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.220624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.221502 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.221502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.222567 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:28.222569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.223435 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.223436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:28.223501 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.223504 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:48:28.223548 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 04:48:28.223581 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":120,"threads_started":1}
I20260810 04:48:28.224120 7852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d10468e8e341c3828479bacd24063d" candidate_uuid: "ca1dd71c489e467384baa66ca23c77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b486ea208abf445380002d9408bcd5d2" is_pre_election: true
I20260810 04:48:28.224165 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d10468e8e341c3828479bacd24063d" candidate_uuid: "ca1dd71c489e467384baa66ca23c77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decc3a782b014ce98153df183d8d0055" is_pre_election: true
I20260810 04:48:28.224277 7852 raft_consensus.cc:2454] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1dd71c489e467384baa66ca23c77aa in term 0.
I20260810 04:48:28.224354 8086 raft_consensus.cc:2454] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca1dd71c489e467384baa66ca23c77aa in term 0.
I20260810 04:48:28.224571 8195 leader_election.cc:302] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca1dd71c489e467384baa66ca23c77aa, decc3a782b014ce98153df183d8d0055; no voters:
I20260810 04:48:28.224753 8416 raft_consensus.cc:2781] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:48:28.224838 8416 raft_consensus.cc:482] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:48:28.224858 8416 raft_consensus.cc:3037] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:28.225766 8416 raft_consensus.cc:504] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.225914 8416 leader_election.cc:288] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [CANDIDATE]: Term 1 election: Requested vote from peers decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969), b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007)
I20260810 04:48:28.226115 8086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d10468e8e341c3828479bacd24063d" candidate_uuid: "ca1dd71c489e467384baa66ca23c77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decc3a782b014ce98153df183d8d0055"
I20260810 04:48:28.226150 7852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d10468e8e341c3828479bacd24063d" candidate_uuid: "ca1dd71c489e467384baa66ca23c77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b486ea208abf445380002d9408bcd5d2"
I20260810 04:48:28.226208 8086 raft_consensus.cc:3037] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:28.226251 7852 raft_consensus.cc:3037] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:28.226955 8086 raft_consensus.cc:2454] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1dd71c489e467384baa66ca23c77aa in term 1.
I20260810 04:48:28.226985 7852 raft_consensus.cc:2454] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca1dd71c489e467384baa66ca23c77aa in term 1.
I20260810 04:48:28.227170 8195 leader_election.cc:302] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b486ea208abf445380002d9408bcd5d2, ca1dd71c489e467384baa66ca23c77aa; no voters:
I20260810 04:48:28.227283 8416 raft_consensus.cc:2781] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:28.227450 8416 raft_consensus.cc:686] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 1 LEADER]: Becoming Leader. State: Replica: ca1dd71c489e467384baa66ca23c77aa, State: Running, Role: LEADER
I20260810 04:48:28.227530 8416 consensus_queue.cc:237] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.228411 7606 catalog_manager.cc:5495] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa reported cstate change: term changed from 0 to 1, leader changed from <none> to ca1dd71c489e467384baa66ca23c77aa (127.14.237.3). New cstate: current_term: 1 leader_uuid: "ca1dd71c489e467384baa66ca23c77aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } health_report { overall_health: UNKNOWN } } }
I20260810 04:48:28.251181 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.249178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.14.237.3:32963)
0810 04:48:28.249260 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:28.249262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:28.249273 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:48:28.249305 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:28.249306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:28.249325 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:28.249400 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:48:28.249521 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.249936 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.250562 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:28.250738 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:28.250786 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:28.250914 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:48:28.251000 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:48:28.251055 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:28.251096 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 04:48:28.251111 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 04:48:28.252934 7852 raft_consensus.cc:1261] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Refusing update from remote peer ca1dd71c489e467384baa66ca23c77aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:28.252959 8086 raft_consensus.cc:1261] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 1 FOLLOWER]: Refusing update from remote peer ca1dd71c489e467384baa66ca23c77aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:28.253285 8416 consensus_queue.cc:1035] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:28.253432 8422 consensus_queue.cc:1035] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:28.255844 8428 mvcc.cc:205] Tried to move back new op lower bound from 7316837614601519104 to 7316837614500655104. Current Snapshot: MvccSnapshot[applied={T|T < 7316837614601519104}]
I20260810 04:48:28.255863 8429 mvcc.cc:205] Tried to move back new op lower bound from 7316837614601519104 to 7316837614500655104. Current Snapshot: MvccSnapshot[applied={T|T < 7316837614601519104}]
I20260810 04:48:28.258260 7606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:48:28.258363 7606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:48:28.259052 7606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Sending DeleteTablet for 3 replicas of tablet 87d10468e8e341c3828479bacd24063d
I20260810 04:48:28.259346 8275 tablet_service.cc:1514] Processing DeleteTablet for tablet 87d10468e8e341c3828479bacd24063d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:28 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:37264
I20260810 04:48:28.259438 8053 tablet_service.cc:1514] Processing DeleteTablet for tablet 87d10468e8e341c3828479bacd24063d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:28 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:57596
I20260810 04:48:28.259522 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 7679
I20260810 04:48:28.259584 7821 tablet_service.cc:1514] Processing DeleteTablet for tablet 87d10468e8e341c3828479bacd24063d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:28 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:56082
I20260810 04:48:28.259732 8447 tablet_replica.cc:330] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: stopping tablet replica
I20260810 04:48:28.259858 8447 raft_consensus.cc:2229] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:48:28.259972 8447 raft_consensus.cc:2258] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:28.265594 8451 tablet_replica.cc:330] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: stopping tablet replica
I20260810 04:48:28.265704 8451 raft_consensus.cc:2229] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:48:28.265812 8451 raft_consensus.cc:2258] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:28.267202 8451 ts_tablet_manager.cc:1905] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:28.268882 8447 ts_tablet_manager.cc:1905] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:28.269327 8451 ts_tablet_manager.cc:1918] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:28.269409 8451 log.cc:1196] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/87d10468e8e341c3828479bacd24063d
I20260810 04:48:28.269703 8451 ts_tablet_manager.cc:1939] T 87d10468e8e341c3828479bacd24063d P decc3a782b014ce98153df183d8d0055: Deleting consensus metadata
I20260810 04:48:28.270637 8447 ts_tablet_manager.cc:1918] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:28.270747 8447 log.cc:1196] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/87d10468e8e341c3828479bacd24063d
I20260810 04:48:28.270818 7571 catalog_manager.cc:4841] TS decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969): tablet 87d10468e8e341c3828479bacd24063d (table test-table [id=ddc87b75aef44eb8a54f0f973e486981]) successfully deleted
I20260810 04:48:28.271044 8447 ts_tablet_manager.cc:1939] T 87d10468e8e341c3828479bacd24063d P ca1dd71c489e467384baa66ca23c77aa: Deleting consensus metadata
I20260810 04:48:28.271430 7571 catalog_manager.cc:4841] TS ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963): tablet 87d10468e8e341c3828479bacd24063d (table test-table [id=ddc87b75aef44eb8a54f0f973e486981]) successfully deleted
W20260810 04:48:28.276142 7571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:43007 (error 108)
I20260810 04:48:28.276474 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 7929
I20260810 04:48:28.277096 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.276573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43007 (local address 127.0.0.1:56098)
0810 04:48:28.277017 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.277056 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423}
W20260810 04:48:28.278549 7571 catalog_manager.cc:4584] TS b486ea208abf445380002d9408bcd5d2 (127.14.237.1:43007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87d10468e8e341c3828479bacd24063d: Network error: Client connection negotiation failed: client connection to 127.14.237.1:43007: connect: Connection refused (error 111)
I20260810 04:48:28.285061 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 8166
I20260810 04:48:28.294497 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 7541
I20260810 04:48:28.303036 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35221
--webserver_interface=127.14.237.62
--webserver_port=38005
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:35221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:28.387434 8456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:28.387632 8456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:28.387653 8456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:28.388775 8456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:28.388818 8456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:28.388832 8456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:28.388878 8456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:28.388895 8456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:28.390412 8456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:35221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:28.390668 8456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:28.390906 8456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:28.395783 8465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:28.395776 8464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:28.395776 8467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:28.396232 8456 server_base.cc:1034] running on GCE node
I20260810 04:48:28.396503 8456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:28.396777 8456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:28.397913 8456 hybrid_clock.cc:648] HybridClock initialized: now 1786337308397892 us; error 36 us; skew 500 ppm
Aug 10 04:48:28 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:28.400441 8456 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:48:28.400908 8456 webserver.cc:460] Webserver started at http://127.14.237.62:38005/ using document root <none> and password file <none>
I20260810 04:48:28.401088 8456 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:28.401129 8456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:28.402428 8456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:28.403100 8476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.403307 8456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:28.403383 8456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "ef3c616bf972499888b80f39d3b13619"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "065ee9eebf3dd1de65fd395d03e5f2d9"
server_key_iv: "06a7c89ae6f62dc3e1b320d7dd5ebee0"
server_key_version: "encryptionkey@0"
I20260810 04:48:28.403687 8456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:28.414830 8456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:28.416520 8456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:28.416628 8456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:28.420698 8456 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:35221
I20260810 04:48:28.420732 8578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:35221 every 8 connection(s)
I20260810 04:48:28.421111 8456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:48:28.422648 8579 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:48:28.423328 8579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Bootstrap starting.
I20260810 04:48:28.428413 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 8456
I20260810 04:48:28.428797 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:43007
--local_ip_for_outbound_sockets=127.14.237.1
--tserver_master_addrs=127.14.237.62:35221
--webserver_port=38553
--webserver_interface=127.14.237.1
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:48:28.432613 8579 log.cc:824] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:28.434935 8579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:48:28.435279 8579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619: Bootstrap complete.
I20260810 04:48:28.437667 8579 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:28.437881 8579 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3c616bf972499888b80f39d3b13619, State: Initialized, Role: FOLLOWER
I20260810 04:48:28.438043 8579 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:28.438112 8579 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:48:28.438138 8579 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:48:28.438205 8579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 2 FOLLOWER]: Advancing to term 3
I20260810 04:48:28.439189 8579 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:28.439277 8579 leader_election.cc:302] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef3c616bf972499888b80f39d3b13619; no voters:
I20260810 04:48:28.439467 8579 leader_election.cc:288] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 04:48:28.439513 8585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 04:48:28.439643 8585 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [term 3 LEADER]: Becoming Leader. State: Replica: ef3c616bf972499888b80f39d3b13619, State: Running, Role: LEADER
I20260810 04:48:28.439746 8579 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:28.439740 8585 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } }
I20260810 04:48:28.440152 8587 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef3c616bf972499888b80f39d3b13619. Latest consensus state: current_term: 3 leader_uuid: "ef3c616bf972499888b80f39d3b13619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } } }
I20260810 04:48:28.440235 8587 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:28.440479 8590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:48:28.440919 8586 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ef3c616bf972499888b80f39d3b13619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c616bf972499888b80f39d3b13619" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35221 } } }
I20260810 04:48:28.441011 8586 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3c616bf972499888b80f39d3b13619 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:28.441344 8590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:48:28.441455 8590 catalog_manager.cc:1259] Loaded cluster ID: 81eba298d3ba4674bb29b079b8b5abcd
I20260810 04:48:28.441664 8590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 04:48:28.442008 8590 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb4b2a019ad google::LogMessage::Fail() at ??:0
@ 0x7fb4b2a0488a google::LogMessage::SendToLog() at ??:0
@ 0x7fb4b2a014de google::LogMessage::Flush() at ??:0
@ 0x7fb4b2a02a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb4ba9220f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb4ba92b8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb4b48cf0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb4b48c4954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb4b57286db start_thread at ??:0
@ 0x7fb4b14ff61f clone at ??:0
W20260810 04:48:28.525874 8581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:28.526072 8581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:28.526094 8581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:28.527252 8581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:28.527293 8581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:28.527318 8581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:48:28.528750 8581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:43007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=38553
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:28.528998 8581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:28.529302 8581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:28.534377 8636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:28.534385 8634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:28.534387 8638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:28.535215 8581 server_base.cc:1034] running on GCE node
I20260810 04:48:28.535465 8581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:28.535718 8581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:28.536875 8581 hybrid_clock.cc:648] HybridClock initialized: now 1786337308536851 us; error 39 us; skew 500 ppm
Aug 10 04:48:28 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:28.539695 8581 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:48:28.540166 8581 webserver.cc:460] Webserver started at http://127.14.237.1:38553/ using document root <none> and password file <none>
I20260810 04:48:28.540413 8581 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:28.540481 8581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:28.541922 8581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:28.542644 8648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.542842 8581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:28.542948 8581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "b486ea208abf445380002d9408bcd5d2"
format_stamp: "Formatted at 2026-08-10 04:48:26 on dist-test-slave-8hhm"
server_key: "2bcca5e21bc56510f8154e61c31658e7"
server_key_iv: "7698ad1b97e21a8204ea87714dca89fd"
server_key_version: "encryptionkey@0"
I20260810 04:48:28.543303 8581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:28.566129 8581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:28.568324 8581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:28.568485 8581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:28.568720 8581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:28.569144 8656 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:48:28.571189 8581 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:48:28.571270 8581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:48:28.571321 8581 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:48:28.572002 8581 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:48:28.572060 8581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:48:28.572153 8656 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Bootstrap starting.
I20260810 04:48:28.579130 8581 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:43007
I20260810 04:48:28.579470 8581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:48:28.580085 8843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:43007 every 8 connection(s)
I20260810 04:48:28.582823 8656 log.cc:824] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:28.585319 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 8581
I20260810 04:48:28.585383 8656 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:48:28.585614 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:37969
--local_ip_for_outbound_sockets=127.14.237.2
--tserver_master_addrs=127.14.237.62:35221
--webserver_port=33563
--webserver_interface=127.14.237.2
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:48:28.585696 8656 tablet_bootstrap.cc:492] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Bootstrap complete.
I20260810 04:48:28.586889 8656 ts_tablet_manager.cc:1397] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260810 04:48:28.588238 8656 raft_consensus.cc:348] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.588564 8656 raft_consensus.cc:729] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b486ea208abf445380002d9408bcd5d2, State: Initialized, Role: FOLLOWER
I20260810 04:48:28.588698 8656 consensus_queue.cc:260] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:28.588970 8656 ts_tablet_manager.cc:1428] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260810 04:48:28.625559 8846 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:48:28.580180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:46397)
0810 04:48:28.580355 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.580397 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:28.580898 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:28.625388 (+ 44490us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":110,"threads_started":1}
W20260810 04:48:28.625860 8844 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:35221 (0 consecutive failures): Network error: Failed to ping master at 127.14.237.62:35221: Client connection negotiation failed: client connection to 127.14.237.62:35221: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:48:28.626233 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.626115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:58841)
0810 04:48:28.626169 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.626212 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:48:28.626667 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.626577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:42235)
0810 04:48:28.626632 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.626653 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 04:48:28.626780 8844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:48:28.670029 8849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:28.670252 8849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:28.670291 8849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:28.671391 8849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:28.671454 8849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:28.671499 8849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:48:28.673038 8849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:37969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=33563
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:28.673291 8849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:28.673528 8849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:28.678282 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:28.678328 8849 server_base.cc:1034] running on GCE node
W20260810 04:48:28.678251 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:28.678252 8862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:28.678684 8849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:28.678920 8849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:28.680085 8849 hybrid_clock.cc:648] HybridClock initialized: now 1786337308680069 us; error 28 us; skew 500 ppm
Aug 10 04:48:28 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:28.682725 8849 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:48:28.683211 8849 webserver.cc:460] Webserver started at http://127.14.237.2:33563/ using document root <none> and password file <none>
I20260810 04:48:28.683427 8849 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:28.683472 8849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:28.684721 8849 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:48:28.685418 8879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.685622 8849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:48:28.685702 8849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "decc3a782b014ce98153df183d8d0055"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "7aa57745e71bc2abd56be2c57c7c3e8e"
server_key_iv: "85c22433f9daf927fbff8729886370c3"
server_key_version: "encryptionkey@0"
I20260810 04:48:28.686038 8849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:28.700194 8849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:28.701960 8849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:28.702112 8849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:28.702317 8849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:28.702622 8849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:28.702675 8849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.702708 8849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:28.702733 8849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.708892 8849 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:37969
I20260810 04:48:28.708940 9060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:37969 every 8 connection(s)
I20260810 04:48:28.709312 8849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:48:28.710317 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.709964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:46543)
0810 04:48:28.710116 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.710188 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
W20260810 04:48:28.710722 9061 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:35221 (0 consecutive failures): Network error: Failed to ping master at 127.14.237.62:35221: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
I20260810 04:48:28.711014 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.710921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:33425)
0810 04:48:28.710965 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.710992 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 04:48:28.711344 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.711253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:60193)
0810 04:48:28.711288 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.711328 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 04:48:28.711472 9061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:48:28.711938 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 8849
I20260810 04:48:28.712205 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:32963
--local_ip_for_outbound_sockets=127.14.237.3
--tserver_master_addrs=127.14.237.62:35221
--webserver_port=35401
--webserver_interface=127.14.237.3
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:28.793378 9068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:28.793604 9068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:28.793642 9068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:28.794778 9068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:28.794842 9068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:28.794886 9068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:48:28.796343 9068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:32963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=35401
--tserver_master_addrs=127.14.237.62:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:28.796613 9068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:28.796839 9068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:28.801826 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:28.801805 9076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:28.801831 9068 server_base.cc:1034] running on GCE node
W20260810 04:48:28.801805 9075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:28.802206 9068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:28.802397 9068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:28.803567 9068 hybrid_clock.cc:648] HybridClock initialized: now 1786337308803550 us; error 29 us; skew 500 ppm
Aug 10 04:48:28 dist-test-slave-8hhm krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:28.806105 9068 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:48:28.806558 9068 webserver.cc:460] Webserver started at http://127.14.237.3:35401/ using document root <none> and password file <none>
I20260810 04:48:28.806730 9068 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:28.806772 9068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:28.808039 9068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:28.808632 9088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.808825 9068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:28.808911 9068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "ca1dd71c489e467384baa66ca23c77aa"
format_stamp: "Formatted at 2026-08-10 04:48:27 on dist-test-slave-8hhm"
server_key: "59fa9dcc113a2aa0d3e1239da52aa402"
server_key_iv: "ec3158528a423432d60c5ef9992e2d52"
server_key_version: "encryptionkey@0"
I20260810 04:48:28.809281 9068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:28.852625 9068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:28.854521 9068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:28.854655 9068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:28.854862 9068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:28.855213 9068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:28.855252 9068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.855316 9068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:28.855350 9068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:28.861428 9068 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:32963
I20260810 04:48:28.861478 9284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:32963 every 8 connection(s)
I20260810 04:48:28.861805 9068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:48:28.862792 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.862423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:33547)
0810 04:48:28.862572 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.862650 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
W20260810 04:48:28.863229 9285 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:35221 (0 consecutive failures): Network error: Failed to ping master at 127.14.237.62:35221: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
I20260810 04:48:28.863492 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.863396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:43721)
0810 04:48:28.863450 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.863476 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 04:48:28.863745 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:28.863669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:49611)
0810 04:48:28.863704 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:28.863732 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 04:48:28.863870 9285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:48:28.868251 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 9068
I20260810 04:48:29.627410 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:29.627176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:36451)
0810 04:48:29.627317 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:29.627353 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260810 04:48:29.712007 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:29.711771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:48773)
0810 04:48:29.711923 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:29.711958 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260810 04:48:29.864390 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:29.864189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:45001)
0810 04:48:29.864304 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:29.864344 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260810 04:48:29.923600 9294 raft_consensus.cc:482] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:29.923774 9294 raft_consensus.cc:504] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:29.924130 9294 leader_election.cc:288] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963), decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969)
Aug 10 04:48:29 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.2@KRBTEST.COM
Aug 10 04:48:29 dist-test-slave-8hhm krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337308, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.3@KRBTEST.COM
I20260810 04:48:29.935674 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:29.924465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:58539 (local address 127.14.237.3:32963)
0810 04:48:29.924633 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:29.924646 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:29.924687 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:48:29.925160 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:29.925162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:29.925204 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:29.925416 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:29.926175 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:29.927116 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:29.928199 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:29.928626 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:29.928745 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:29.931637 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:29.931640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:29.931640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:29.931651 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:29.933669 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:29.934166 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:29.934170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:29.934170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:29.934307 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:29.934645 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:29.934645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:29.934646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:29.934967 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:29.935058 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:29.935419 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 04:48:29.935559 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260810 04:48:29.935674 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:29.924583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:56587 (local address 127.14.237.2:37969)
0810 04:48:29.924762 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:29.924771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:29.924799 (+ 28us) server_negotiation.cc:407] Connection header received
0810 04:48:29.925208 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:29.925271 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:29.925321 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:29.925463 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:29.926201 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:29.927128 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:29.928191 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:29.928612 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:29.928740 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:29.931241 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:29.931245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:29.931245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:29.931255 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:29.933383 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:29.934166 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:29.934170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:29.934170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:29.934310 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:29.934644 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:29.934645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:29.934645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:29.934967 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:29.935058 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:29.935419 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 04:48:29.935559 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260810 04:48:29.935694 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:29.924344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:32963 (local address 127.14.237.1:58539)
0810 04:48:29.924421 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:29.924437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:29.924513 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:29.925460 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:29.925464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:29.925849 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:29.926034 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:29.926041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:29.927242 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:29.927243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:29.928067 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:29.928069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:29.928720 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:29.928721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:29.928845 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:29.929512 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:29.929520 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:29.931516 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:29.933748 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:29.933749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:29.933750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:29.934082 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:29.934393 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:29.934393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:29.934394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:29.934567 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:29.935048 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:29.935056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:29.935216 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 04:48:29.935346 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 04:48:29.935546 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2}
I20260810 04:48:29.935707 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:29.924482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:37969 (local address 127.14.237.1:56587)
0810 04:48:29.924623 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:29.924631 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:29.924661 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:29.925469 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:29.925470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:29.925851 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:29.926033 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:29.926039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:29.927282 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:29.927283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:29.928068 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:29.928070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:29.928720 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:29.928721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:29.928842 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:29.929563 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:29.929567 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:29.931112 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:29.933521 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:29.933529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:29.933532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:29.934056 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:29.934393 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:29.934393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:29.934394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:29.934567 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:29.935048 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:29.935056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:29.935216 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 04:48:29.935341 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 04:48:29.935606 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":181,"thread_start_us":53,"threads_started":1}
W20260810 04:48:29.936758 8649 leader_election.cc:341] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969): Not found: Tablet not found: 87d10468e8e341c3828479bacd24063d
W20260810 04:48:29.936882 8649 leader_election.cc:341] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963): Not found: Tablet not found: 87d10468e8e341c3828479bacd24063d
I20260810 04:48:29.936934 8649 leader_election.cc:302] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b486ea208abf445380002d9408bcd5d2; no voters: ca1dd71c489e467384baa66ca23c77aa, decc3a782b014ce98153df183d8d0055
I20260810 04:48:29.937034 9294 raft_consensus.cc:2726] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:48:30.628067 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:30.627865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:53253)
0810 04:48:30.627985 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:30.628019 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260810 04:48:30.712616 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:30.712426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:59941)
0810 04:48:30.712541 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:30.712573 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260810 04:48:30.864991 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:30.864810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:54769)
0810 04:48:30.864921 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:30.864954 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260810 04:48:31.628880 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:31.628639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:36159)
0810 04:48:31.628788 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:31.628825 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260810 04:48:31.713195 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:31.713011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:42105)
0810 04:48:31.713127 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:31.713161 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260810 04:48:31.865809 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:31.865586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:51845)
0810 04:48:31.865737 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:31.865772 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260810 04:48:32.149698 9313 raft_consensus.cc:482] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:32.149801 9313 raft_consensus.cc:504] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:32.149952 9313 leader_election.cc:288] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963), decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969)
W20260810 04:48:32.150383 8649 leader_election.cc:341] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963): Not found: Tablet not found: 87d10468e8e341c3828479bacd24063d
W20260810 04:48:32.150460 8649 leader_election.cc:341] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969): Not found: Tablet not found: 87d10468e8e341c3828479bacd24063d
I20260810 04:48:32.150475 8649 leader_election.cc:302] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b486ea208abf445380002d9408bcd5d2; no voters: ca1dd71c489e467384baa66ca23c77aa, decc3a782b014ce98153df183d8d0055
I20260810 04:48:32.150547 9313 raft_consensus.cc:2726] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:48:32.629756 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:32.629433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:58159)
0810 04:48:32.629672 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:32.629710 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":190,"threads_started":1}
I20260810 04:48:32.713841 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:32.713648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:49691)
0810 04:48:32.713771 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:32.713805 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260810 04:48:32.866583 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:32.866369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:38501)
0810 04:48:32.866509 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:32.866543 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260810 04:48:33.630564 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:33.630294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:54421)
0810 04:48:33.630482 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:33.630520 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260810 04:48:33.714459 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:33.714267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:47081)
0810 04:48:33.714388 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:33.714421 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260810 04:48:33.867378 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:33.867116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:42539)
0810 04:48:33.867301 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:33.867338 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":140,"threads_started":1}
I20260810 04:48:33.867815 9324 raft_consensus.cc:482] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:33.867915 9324 raft_consensus.cc:504] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:33.868091 9324 leader_election.cc:288] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963), decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969)
W20260810 04:48:33.868487 8649 leader_election.cc:341] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963): Not found: Tablet not found: 87d10468e8e341c3828479bacd24063d
W20260810 04:48:33.868595 8649 leader_election.cc:341] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969): Not found: Tablet not found: 87d10468e8e341c3828479bacd24063d
I20260810 04:48:33.868642 8649 leader_election.cc:302] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b486ea208abf445380002d9408bcd5d2; no voters: ca1dd71c489e467384baa66ca23c77aa, decc3a782b014ce98153df183d8d0055
I20260810 04:48:33.868741 9324 raft_consensus.cc:2726] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:48:34.631305 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:34.631074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:44803)
0810 04:48:34.631214 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:34.631259 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260810 04:48:34.715200 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:34.714960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:36445)
0810 04:48:34.715107 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:34.715154 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260810 04:48:34.868079 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:34.867843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:38859)
0810 04:48:34.867981 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:34.868018 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260810 04:48:35.632064 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:35.631840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:50005)
0810 04:48:35.631979 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:35.632018 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260810 04:48:35.715847 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:35.715627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:38529)
0810 04:48:35.715774 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:35.715810 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260810 04:48:35.868921 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:35.868692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:52731)
0810 04:48:35.868841 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:35.868876 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260810 04:48:36.632746 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:36.632524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:38499)
0810 04:48:36.632659 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:36.632701 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260810 04:48:36.716460 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:36.716266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:55251)
0810 04:48:36.716389 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:36.716422 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260810 04:48:36.869679 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:36.869466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:57361)
0810 04:48:36.869601 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:36.869636 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260810 04:48:37.633615 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:37.633389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:53037)
0810 04:48:37.633525 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:37.633562 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260810 04:48:37.717307 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:37.716984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:51803)
0810 04:48:37.717145 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:37.717184 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
I20260810 04:48:37.870340 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:37.870148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:38715)
0810 04:48:37.870264 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:37.870298 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260810 04:48:38.349921 9341 raft_consensus.cc:482] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:38.350052 9341 raft_consensus.cc:504] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1dd71c489e467384baa66ca23c77aa" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 32963 } } peers { permanent_uuid: "decc3a782b014ce98153df183d8d0055" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 37969 } } peers { permanent_uuid: "b486ea208abf445380002d9408bcd5d2" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43007 } }
I20260810 04:48:38.350325 9341 leader_election.cc:288] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963), decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969)
W20260810 04:48:38.350724 8649 leader_election.cc:341] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ca1dd71c489e467384baa66ca23c77aa (127.14.237.3:32963): Not found: Tablet not found: 87d10468e8e341c3828479bacd24063d
W20260810 04:48:38.350809 8649 leader_election.cc:341] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer decc3a782b014ce98153df183d8d0055 (127.14.237.2:37969): Not found: Tablet not found: 87d10468e8e341c3828479bacd24063d
I20260810 04:48:38.350826 8649 leader_election.cc:302] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b486ea208abf445380002d9408bcd5d2; no voters: ca1dd71c489e467384baa66ca23c77aa, decc3a782b014ce98153df183d8d0055
I20260810 04:48:38.350935 9341 raft_consensus.cc:2726] T 87d10468e8e341c3828479bacd24063d P b486ea208abf445380002d9408bcd5d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:48:38.634346 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:38.634087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:43559)
0810 04:48:38.634264 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:38.634300 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260810 04:48:38.718010 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:38.717778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:33453)
0810 04:48:38.717922 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:38.717961 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260810 04:48:38.871006 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:38.870779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:44151)
0810 04:48:38.870916 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:38.870957 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260810 04:48:39.635057 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:39.634803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:44313)
0810 04:48:39.634969 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:39.635011 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260810 04:48:39.718750 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:39.718546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:52887)
0810 04:48:39.718669 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:39.718704 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260810 04:48:39.871757 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:39.871547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:33537)
0810 04:48:39.871671 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:39.871704 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260810 04:48:40.635766 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:40.635549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:35789)
0810 04:48:40.635678 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:40.635720 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260810 04:48:40.719540 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:40.719341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:39247)
0810 04:48:40.719466 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:40.719499 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260810 04:48:40.872483 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:40.872240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:36117)
0810 04:48:40.872373 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:40.872416 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260810 04:48:41.636485 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:41.636234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:38999)
0810 04:48:41.636401 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:41.636440 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260810 04:48:41.720382 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:41.720035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:57759)
0810 04:48:41.720297 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:41.720337 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
I20260810 04:48:41.873152 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:41.872926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:46501)
0810 04:48:41.873078 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:41.873109 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260810 04:48:42.637174 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:42.636941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:51263)
0810 04:48:42.637089 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:42.637128 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260810 04:48:42.721174 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:42.720877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:49173)
0810 04:48:42.721095 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:42.721130 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260810 04:48:42.873806 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:42.873611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:35849)
0810 04:48:42.873730 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:42.873764 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260810 04:48:43.637952 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:43.637709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.1:60759)
0810 04:48:43.637862 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:43.637905 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260810 04:48:43.721961 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:43.721749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.2:57149)
0810 04:48:43.721883 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:43.721917 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260810 04:48:43.874334 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 8581
I20260810 04:48:43.881698 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:43.874318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:35221 (local address 127.14.237.3:44307)
0810 04:48:43.881576 (+ 7258us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:43.881634 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7233,"thread_start_us":77,"threads_started":1}
I20260810 04:48:43.888388 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 8849
I20260810 04:48:43.899598 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 9068
2026-08-10T04:48:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18555 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:48:43 dist-test-slave-8hhm krb5kdc[9378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:48:43 dist-test-slave-8hhm krb5kdc[9378](info): set up 2 sockets
Aug 10 04:48:43 dist-test-slave-8hhm krb5kdc[9378](info): commencing operation
krb5kdc: starting...
W20260810 04:48:45.158480 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:48:45.280025 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38863
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:38863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:45.362478 9399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:45.362674 9399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:45.362696 9399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:45.363759 9399 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:48:45.363791 9399 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:48:45.363804 9399 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:48:45.363816 9399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:48:45.363828 9399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:45.363842 9399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:45.363860 9399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:48:45.363873 9399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:48:45.365361 9399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:38863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:45.365548 9399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:45.365707 9399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:45.370857 9406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:45.370831 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:45.370841 9415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:45.370858 9399 server_base.cc:1034] running on GCE node
I20260810 04:48:45.371466 9399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:45.371737 9399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:45.372885 9399 hybrid_clock.cc:648] HybridClock initialized: now 1786337325372872 us; error 33 us; skew 500 ppm
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:45.375411 9399 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:48:45.375883 9399 webserver.cc:460] Webserver started at http://127.14.237.62:39711/ using document root <none> and password file <none>
I20260810 04:48:45.376112 9399 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:45.376157 9399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:45.376312 9399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:45.377461 9399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "5969816e2a4a4d6dae31219b44ed25e7"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b30ec9bef3ea3c9f31070bbf43495924"
server_key_iv: "5f5d685d14083f7beac467011b45afdc"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.377839 9399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "5969816e2a4a4d6dae31219b44ed25e7"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b30ec9bef3ea3c9f31070bbf43495924"
server_key_iv: "5f5d685d14083f7beac467011b45afdc"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.379226 9399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:48:45.379967 9425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.380167 9399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:45.380281 9399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "5969816e2a4a4d6dae31219b44ed25e7"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b30ec9bef3ea3c9f31070bbf43495924"
server_key_iv: "5f5d685d14083f7beac467011b45afdc"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.380360 9399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:45.400606 9399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:45.402593 9399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:45.402774 9399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:45.406664 9399 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:38863
I20260810 04:48:45.406708 9508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:38863 every 8 connection(s)
I20260810 04:48:45.407080 9399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:48:45.407965 9509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:45.410336 9509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Bootstrap starting.
I20260810 04:48:45.410933 9509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:45.411238 9509 log.cc:824] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:45.411932 9509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: No bootstrap required, opened a new log
I20260810 04:48:45.413566 9509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:48:45.413727 9509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:45.413790 9509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5969816e2a4a4d6dae31219b44ed25e7, State: Initialized, Role: FOLLOWER
I20260810 04:48:45.413932 9509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:48:45.413995 9509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:48:45.414036 9509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:48:45.414088 9509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:45.414642 9509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:48:45.414772 9509 leader_election.cc:302] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5969816e2a4a4d6dae31219b44ed25e7; no voters:
I20260810 04:48:45.414947 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 9399
I20260810 04:48:45.414983 9509 leader_election.cc:288] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:48:45.415015 9512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:45.415062 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:48:45.415226 9512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 1 LEADER]: Becoming Leader. State: Replica: 5969816e2a4a4d6dae31219b44ed25e7, State: Running, Role: LEADER
I20260810 04:48:45.415287 15284 external_mini_cluster.cc:1428] Setting key 9924e394d9c016b51b2d21956963730e
I20260810 04:48:45.415243 9509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:48:45.415380 9512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:48:45.416162 9517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5969816e2a4a4d6dae31219b44ed25e7. Latest consensus state: current_term: 1 leader_uuid: "5969816e2a4a4d6dae31219b44ed25e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } } }
I20260810 04:48:45.416277 9517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:45.416479 9529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:48:45.416580 9515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5969816e2a4a4d6dae31219b44ed25e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } } }
I20260810 04:48:45.416683 9515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: This master's current role is: LEADER
I20260810 04:48:45.417104 9529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:48:45.418905 9529 catalog_manager.cc:1347] Generated new cluster ID: 3735365d8e0b4ec6beddde6322358dac
I20260810 04:48:45.418958 9529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:45.428007 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.415789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.14.237.62:38863)
0810 04:48:45.415977 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:45.415989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:45.416088 (+ 99us) server_negotiation.cc:407] Connection header received
0810 04:48:45.416990 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:45.417000 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:45.417071 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:45.417180 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:45.418189 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.419544 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.420636 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.421251 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.421350 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.424219 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:45.424225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:45.424226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:45.424243 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:45.426314 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.426724 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.426727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.426727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.426852 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.427065 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.427066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.427067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.427378 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:45.427523 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:45.427633 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 04:48:45.427778 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260810 04:48:45.452728 9529 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:48:45.453598 9529 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:48:45.462088 9529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Generated new TSK 0
I20260810 04:48:45.462308 9529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:48:45.499857 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38863
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:45.587064 9548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:45.587275 9548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:45.587301 9548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:45.588444 9548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:45.588488 9548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:45.588537 9548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:48:45.590019 9548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:45.590281 9548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:45.590512 9548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:48:45.595798 9548 server_base.cc:1034] running on GCE node
W20260810 04:48:45.595768 9559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:45.595757 9558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:45.595801 9565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:45.596319 9548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:45.596586 9548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:45.597764 9548 hybrid_clock.cc:648] HybridClock initialized: now 1786337325597758 us; error 35 us; skew 500 ppm
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:45.600339 9548 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:48:45.600762 9548 webserver.cc:460] Webserver started at http://127.14.237.1:41591/ using document root <none> and password file <none>
I20260810 04:48:45.600992 9548 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:45.601063 9548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:45.601262 9548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:45.602375 9548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "c93f1f1262444fb78d335310a66b7b45"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "547db94a2da0127c91c7ce06aa2df66b"
server_key_iv: "a2a3ed274ad3d086a66bde7af70e8bec"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.602788 9548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "c93f1f1262444fb78d335310a66b7b45"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "547db94a2da0127c91c7ce06aa2df66b"
server_key_iv: "a2a3ed274ad3d086a66bde7af70e8bec"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.604133 9548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:48:45.604812 9575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.605021 9548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:48:45.605132 9548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "c93f1f1262444fb78d335310a66b7b45"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "547db94a2da0127c91c7ce06aa2df66b"
server_key_iv: "a2a3ed274ad3d086a66bde7af70e8bec"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.605214 9548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:45.612378 9548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:45.614172 9548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:45.614331 9548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:45.614529 9548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:45.614833 9548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:45.614897 9548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.614934 9548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:45.614961 9548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.621007 9548 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:43207
I20260810 04:48:45.621078 9764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:43207 every 8 connection(s)
I20260810 04:48:45.621397 9548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:48:45.625967 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 9548
I20260810 04:48:45.626075 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:48:45.626286 15284 external_mini_cluster.cc:1428] Setting key 7e579360078a3856bbede42c8007dc41
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:45.630925 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.622102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:42197)
0810 04:48:45.622281 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:45.622314 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:45.622770 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:45.622961 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:45.622964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:45.623502 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:45.623673 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.623678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.624340 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.624341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.625095 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.625097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.625500 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.625501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.625595 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.626046 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:45.626058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:45.627663 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:45.629484 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:45.629492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:45.629493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:45.629721 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:45.629956 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:45.629957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:45.629957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:45.630069 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:45.630379 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:45.630386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:45.630532 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 04:48:45.630656 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 04:48:45.630776 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260810 04:48:45.630939 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.622125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:42197 (local address 127.14.237.62:38863)
0810 04:48:45.622199 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:45.622201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:45.622395 (+ 194us) server_negotiation.cc:407] Connection header received
0810 04:48:45.622849 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:45.622850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:45.622860 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:45.622911 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:45.623768 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.624260 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.625190 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.625429 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.625479 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.627764 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:45.627766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:45.627767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:45.627774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:45.629372 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.629807 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.629808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.629809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.629881 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.630128 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.630129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.630129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.630293 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:45.630365 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:45.630706 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 04:48:45.630804 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:48:45.631474 9765 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38863
I20260810 04:48:45.631585 9765 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:45.631757 9765 heartbeater.cc:511] Master 127.14.237.62:38863 requested a full tablet report, sending...
I20260810 04:48:45.632323 9456 ts_manager.cc:194] Registered new tserver with Master: c93f1f1262444fb78d335310a66b7b45 (127.14.237.1:43207)
I20260810 04:48:45.632911 9456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:42197
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:48:45.653347 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38863
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:45.734994 9771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:45.735179 9771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:45.735203 9771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:45.736254 9771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:45.736286 9771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:45.736311 9771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:48:45.737673 9771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:45.737844 9771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:45.738020 9771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:45.742961 9779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:45.742974 9778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:45.743012 9781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:45.743184 9771 server_base.cc:1034] running on GCE node
I20260810 04:48:45.743440 9771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:45.743667 9771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:45.744848 9771 hybrid_clock.cc:648] HybridClock initialized: now 1786337325744828 us; error 31 us; skew 500 ppm
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:45.747397 9771 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:48:45.747864 9771 webserver.cc:460] Webserver started at http://127.14.237.2:45869/ using document root <none> and password file <none>
I20260810 04:48:45.748039 9771 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:45.748078 9771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:45.748173 9771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:45.749096 9771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "06c35cf4a5de4822badda84d6103959d"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "8064110840cb3532bee667e3467c8f9d"
server_key_iv: "486f0914bb9848e287c863f3f6e86704"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.749496 9771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "06c35cf4a5de4822badda84d6103959d"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "8064110840cb3532bee667e3467c8f9d"
server_key_iv: "486f0914bb9848e287c863f3f6e86704"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.750733 9771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:48:45.751402 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.751556 9771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.751642 9771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "06c35cf4a5de4822badda84d6103959d"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "8064110840cb3532bee667e3467c8f9d"
server_key_iv: "486f0914bb9848e287c863f3f6e86704"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.751699 9771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:45.763073 9771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:45.765286 9771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:45.765443 9771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:45.765686 9771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:45.765987 9771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:45.766021 9771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.766041 9771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:45.766089 9771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.772096 9771 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:40165
I20260810 04:48:45.772167 9970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:40165 every 8 connection(s)
I20260810 04:48:45.772517 9771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:48:45.778033 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 9771
I20260810 04:48:45.778143 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:45.778369 15284 external_mini_cluster.cc:1428] Setting key aa4e3b226ae11f1894cc4dc96c56a5b7
I20260810 04:48:45.781867 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.773299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:37443 (local address 127.14.237.62:38863)
0810 04:48:45.773368 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:45.773370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:45.773576 (+ 206us) server_negotiation.cc:407] Connection header received
0810 04:48:45.774014 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:45.774015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:45.774028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:45.774103 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:45.775031 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.775546 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.776465 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.776638 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.776688 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.778811 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:45.778813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:45.778813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:45.778819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:45.780416 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.780794 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.780795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.780796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.780860 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.781130 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.781131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.781132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.781303 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:45.781377 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:45.781683 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 04:48:45.781780 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:48:45.782073 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.773285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:37443)
0810 04:48:45.773446 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:45.773479 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:45.773927 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:45.774174 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:45.774177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:45.774705 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:45.774871 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.774876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.775637 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.775637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.776364 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.776366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.776708 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.776708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.776812 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.777238 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:45.777246 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:45.778730 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:45.780498 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:45.780504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:45.780506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:45.780720 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:45.780947 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:45.780948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:45.780948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:45.781056 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:45.781393 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:45.781400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:45.781524 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 04:48:45.781658 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 04:48:45.781804 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260810 04:48:45.782549 9974 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38863
I20260810 04:48:45.782649 9974 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:45.782835 9974 heartbeater.cc:511] Master 127.14.237.62:38863 requested a full tablet report, sending...
I20260810 04:48:45.783241 9456 ts_manager.cc:194] Registered new tserver with Master: 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165)
I20260810 04:48:45.783689 9456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:37443
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:48:45.804349 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38863
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:48:45.889266 9979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:48:45.889477 9979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:48:45.889498 9979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:48:45.890522 9979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:48:45.890552 9979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:48:45.890578 9979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:48:45.891942 9979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:48:45.892127 9979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:48:45.892297 9979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:48:45.897028 9993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:48:45.897028 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:45.897063 9979 server_base.cc:1034] running on GCE node
W20260810 04:48:45.897069 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:48:45.897449 9979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:48:45.897696 9979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:48:45.898869 9979 hybrid_clock.cc:648] HybridClock initialized: now 1786337325898853 us; error 27 us; skew 500 ppm
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:48:45.901377 9979 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:48:45.901796 9979 webserver.cc:460] Webserver started at http://127.14.237.3:34393/ using document root <none> and password file <none>
I20260810 04:48:45.902040 9979 fs_manager.cc:359] Metadata directory not provided
I20260810 04:48:45.902107 9979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:48:45.902266 9979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:48:45.903254 9979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "6aed075ee04640f3b0e6aceb2beed670"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b50d2258b9c8262703dd14f4a476a8af"
server_key_iv: "28fb30499ce80210b944651ae06180ff"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.903604 9979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "6aed075ee04640f3b0e6aceb2beed670"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b50d2258b9c8262703dd14f4a476a8af"
server_key_iv: "28fb30499ce80210b944651ae06180ff"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.904912 9979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:48:45.905717 10005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.905884 9979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.905983 9979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "6aed075ee04640f3b0e6aceb2beed670"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b50d2258b9c8262703dd14f4a476a8af"
server_key_iv: "28fb30499ce80210b944651ae06180ff"
server_key_version: "encryptionkey@0"
I20260810 04:48:45.906064 9979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:48:45.920977 9979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:48:45.923326 9979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:48:45.923481 9979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:48:45.923725 9979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:48:45.924021 9979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:48:45.924074 9979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.924116 9979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:48:45.924144 9979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:48:45.930373 9979 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:43865
I20260810 04:48:45.930418 10182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:43865 every 8 connection(s)
I20260810 04:48:45.930701 9979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:48:45 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337325, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:48:45.938989 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 9979
I20260810 04:48:45.939069 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:48:45.939256 15284 external_mini_cluster.cc:1428] Setting key 9f27087293e20c0d29f73ede8e5c8285
I20260810 04:48:45.939718 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.931478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:33877 (local address 127.14.237.62:38863)
0810 04:48:45.931552 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:45.931555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:45.931710 (+ 155us) server_negotiation.cc:407] Connection header received
0810 04:48:45.932217 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:45.932218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:45.932229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:45.932293 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:45.933171 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.933728 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.934657 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.934835 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.934888 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.936980 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:45.936981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:45.936981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:45.936988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:45.938265 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.938609 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.938610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.938610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.938660 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.938935 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.938935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.938935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.939029 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:45.939161 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:45.939527 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 04:48:45.939608 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:48:45.939911 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.931402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:33877)
0810 04:48:45.931577 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:45.931617 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:45.932090 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:45.932363 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:45.932367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:45.932910 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:48:45.933091 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.933095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.933806 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.933807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.934576 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.934578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.934911 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.934912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.935013 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.935504 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:48:45.935513 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:48:45.936894 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:48:45.938336 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:45.938342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:45.938343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:45.938545 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:45.938717 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:48:45.938717 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:48:45.938726 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:48:45.938868 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:48:45.939189 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:48:45.939195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:48:45.939335 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 04:48:45.939477 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 04:48:45.939649 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260810 04:48:45.940405 10183 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38863
I20260810 04:48:45.940491 10183 heartbeater.cc:464] Registering TS with master...
I20260810 04:48:45.940624 10183 heartbeater.cc:511] Master 127.14.237.62:38863 requested a full tablet report, sending...
I20260810 04:48:45.940974 9456 ts_manager.cc:194] Registered new tserver with Master: 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865)
I20260810 04:48:45.941372 9456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:33877
I20260810 04:48:45.951030 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:48:45.957852 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.952280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.14.237.62:38863)
0810 04:48:45.952350 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:45.952352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:45.952417 (+ 65us) server_negotiation.cc:407] Connection header received
0810 04:48:45.952451 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:45.952451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:45.952460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:45.952502 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:48:45.952982 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.953513 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.954150 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.954369 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.954424 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.955640 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:48:45.955641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:48:45.955641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:48:45.955646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:48:45.956823 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.957140 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.957141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.957141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.957190 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:48:45.957416 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:48:45.957417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:48:45.957417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:48:45.957548 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:48:45.957626 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:45.957715 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 04:48:45.957771 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:48:45.959499 9456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:48:45.961027 9456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:48:45.967339 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.963462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.14.237.2:40165)
0810 04:48:45.963984 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:45.963988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:45.963998 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:48:45.964036 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:45.964038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:45.964053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:45.964122 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:45.964323 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.965347 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.966399 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.966908 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.966994 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.967053 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:45.967219 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 04:48:45.967260 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":90,"threads_started":1}
I20260810 04:48:45.967330 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.963206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40165 (local address 127.0.0.1:53952)
0810 04:48:45.963325 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:45.963369 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:45.963435 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:45.964214 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:45.964215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:45.964216 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:45.964248 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.964250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.965460 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.965461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.966315 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.966317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.967008 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.967009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.967089 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.967101 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 04:48:45.967202 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 04:48:45.967262 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260810 04:48:45.967594 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.963501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.14.237.3:43865)
0810 04:48:45.963679 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:45.963684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:45.963732 (+ 48us) server_negotiation.cc:407] Connection header received
0810 04:48:45.963846 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:45.963848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:45.963863 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:45.963949 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:45.964118 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.965178 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.966459 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.967160 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.967232 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.967284 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:45.967487 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 04:48:45.967528 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
I20260810 04:48:45.967715 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.963462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.14.237.1:43207)
0810 04:48:45.963663 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:45.963667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:45.963679 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:48:45.963734 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:45.963737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:45.963753 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:45.963831 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:45.964077 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.964840 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.966272 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.967026 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.967095 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.967150 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:45.967524 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 04:48:45.967569 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260810 04:48:45.967911 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.963335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:43865 (local address 127.0.0.1:48916)
0810 04:48:45.963647 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:45.963659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:45.963744 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:45.963964 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:45.963965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:45.963967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:45.964023 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.964025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.965345 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.965346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.966372 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.966373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.967317 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.967318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.967369 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.967372 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:48:45.967417 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 04:48:45.967449 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":52,"threads_started":1}
I20260810 04:48:45.967449 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:45.963206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43207 (local address 127.0.0.1:56482)
0810 04:48:45.963528 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:45.963538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:45.963578 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:45.963851 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:45.963854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:45.963856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:45.963926 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.963928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.964945 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.964946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.966160 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:45.966163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:45.967188 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:45.967189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:45.967266 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:45.967270 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:48:45.967320 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:48:45.967360 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":4,"thread_start_us":60,"threads_started":1}
I20260810 04:48:45.968573 9865 tablet_service.cc:1467] Processing CreateTablet for tablet 2c62c2ca861444159afba0509a14c16f (DEFAULT_TABLE table=test-table [id=d3eebc41c4e645f885f184ed121b2da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:45.968696 9659 tablet_service.cc:1467] Processing CreateTablet for tablet 2c62c2ca861444159afba0509a14c16f (DEFAULT_TABLE table=test-table [id=d3eebc41c4e645f885f184ed121b2da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:45.968935 9865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c62c2ca861444159afba0509a14c16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:45.969122 9659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c62c2ca861444159afba0509a14c16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:45.969799 10078 tablet_service.cc:1467] Processing CreateTablet for tablet 2c62c2ca861444159afba0509a14c16f (DEFAULT_TABLE table=test-table [id=d3eebc41c4e645f885f184ed121b2da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:48:45.970108 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c62c2ca861444159afba0509a14c16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:48:45.971915 10206 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Bootstrap starting.
I20260810 04:48:45.972393 10212 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Bootstrap starting.
I20260810 04:48:45.972746 10206 tablet_bootstrap.cc:654] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:45.973065 10212 tablet_bootstrap.cc:654] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:45.973277 10214 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: Bootstrap starting.
I20260810 04:48:45.973573 10206 log.cc:824] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:45.973680 10212 log.cc:824] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:45.973951 10214 tablet_bootstrap.cc:654] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: Neither blocks nor log segments found. Creating new log.
I20260810 04:48:45.974308 10206 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: No bootstrap required, opened a new log
I20260810 04:48:45.974391 10206 ts_tablet_manager.cc:1397] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:48:45.974447 10214 log.cc:824] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: Log is configured to *not* fsync() on all Append() calls
I20260810 04:48:45.975087 10214 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: No bootstrap required, opened a new log
I20260810 04:48:45.975138 10214 ts_tablet_manager.cc:1397] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:48:45.975956 10206 raft_consensus.cc:348] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:45.976116 10206 raft_consensus.cc:374] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:45.976163 10206 raft_consensus.cc:729] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93f1f1262444fb78d335310a66b7b45, State: Initialized, Role: FOLLOWER
I20260810 04:48:45.976310 10206 consensus_queue.cc:260] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:45.976619 10206 ts_tablet_manager.cc:1428] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:45.976469 10214 raft_consensus.cc:348] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:45.976689 10214 raft_consensus.cc:374] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:45.976619 9765 heartbeater.cc:503] Master 127.14.237.62:38863 was elected leader, sending a full tablet report...
I20260810 04:48:45.976739 10214 raft_consensus.cc:729] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aed075ee04640f3b0e6aceb2beed670, State: Initialized, Role: FOLLOWER
I20260810 04:48:45.976864 10214 consensus_queue.cc:260] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:45.977319 10214 ts_tablet_manager.cc:1428] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:45.977345 10183 heartbeater.cc:503] Master 127.14.237.62:38863 was elected leader, sending a full tablet report...
I20260810 04:48:45.977418 10212 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: No bootstrap required, opened a new log
I20260810 04:48:45.977469 10212 ts_tablet_manager.cc:1397] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 04:48:45.978974 10212 raft_consensus.cc:348] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:45.979097 10212 raft_consensus.cc:374] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:48:45.979118 10212 raft_consensus.cc:729] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c35cf4a5de4822badda84d6103959d, State: Initialized, Role: FOLLOWER
I20260810 04:48:45.979184 10212 consensus_queue.cc:260] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:45.979391 10212 ts_tablet_manager.cc:1428] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:48:45.979427 9974 heartbeater.cc:503] Master 127.14.237.62:38863 was elected leader, sending a full tablet report...
W20260810 04:48:46.023121 9975 tablet.cc:2369] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:48:46.091027 10220 raft_consensus.cc:482] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:48:46.091239 10220 raft_consensus.cc:504] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:46.091655 10220 leader_election.cc:288] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165), 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865)
I20260810 04:48:46.095695 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:46.092010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:58587 (local address 127.14.237.3:43865)
0810 04:48:46.092052 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:46.092053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:46.092185 (+ 132us) server_negotiation.cc:407] Connection header received
0810 04:48:46.092239 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:46.092240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:46.092255 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:46.092299 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:46.092460 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.093164 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.094780 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.095344 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.095419 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:46.095427 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:46.095586 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 04:48:46.095623 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:48:46.095695 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:46.091954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:45885 (local address 127.14.237.2:40165)
0810 04:48:46.092008 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:46.092010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:46.092040 (+ 30us) server_negotiation.cc:407] Connection header received
0810 04:48:46.092102 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:46.092103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:46.092115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:46.092196 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:48:46.092404 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.093167 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.094775 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.095342 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.095419 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:46.095427 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:46.095587 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 04:48:46.095624 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 04:48:46.095709 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:46.091860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40165 (local address 127.14.237.1:45885)
0810 04:48:46.091979 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:46.091993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:46.092037 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:46.092202 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:46.092203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:46.092205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:46.092316 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:46.092319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.093281 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.093282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:46.094675 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:46.094679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.095428 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.095429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:46.095509 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:46.095511 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:48:46.095566 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:48:46.095613 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":95}
I20260810 04:48:46.095726 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:46.091907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:43865 (local address 127.14.237.1:58587)
0810 04:48:46.092103 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:46.092112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:48:46.092187 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:48:46.092325 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:48:46.092326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:48:46.092328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:48:46.092388 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:46.092389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.093281 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.093282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:46.094676 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:48:46.094679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.095428 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.095429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:48:46.095506 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:46.095511 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:48:46.095567 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:48:46.095613 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":101,"thread_start_us":120,"threads_started":1}
I20260810 04:48:46.096094 9900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c62c2ca861444159afba0509a14c16f" candidate_uuid: "c93f1f1262444fb78d335310a66b7b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c35cf4a5de4822badda84d6103959d" is_pre_election: true
I20260810 04:48:46.096115 10106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c62c2ca861444159afba0509a14c16f" candidate_uuid: "c93f1f1262444fb78d335310a66b7b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed075ee04640f3b0e6aceb2beed670" is_pre_election: true
I20260810 04:48:46.096290 9900 raft_consensus.cc:2454] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c93f1f1262444fb78d335310a66b7b45 in term 0.
I20260810 04:48:46.096303 10106 raft_consensus.cc:2454] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c93f1f1262444fb78d335310a66b7b45 in term 0.
I20260810 04:48:46.096522 9578 leader_election.cc:302] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c35cf4a5de4822badda84d6103959d, c93f1f1262444fb78d335310a66b7b45; no voters:
I20260810 04:48:46.096647 10220 raft_consensus.cc:2781] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:48:46.096710 10220 raft_consensus.cc:482] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:48:46.096732 10220 raft_consensus.cc:3037] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:46.097553 10220 raft_consensus.cc:504] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:46.097695 10220 leader_election.cc:288] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 1 election: Requested vote from peers 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165), 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865)
I20260810 04:48:46.097841 9900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c62c2ca861444159afba0509a14c16f" candidate_uuid: "c93f1f1262444fb78d335310a66b7b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c35cf4a5de4822badda84d6103959d"
I20260810 04:48:46.097886 10106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c62c2ca861444159afba0509a14c16f" candidate_uuid: "c93f1f1262444fb78d335310a66b7b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed075ee04640f3b0e6aceb2beed670"
I20260810 04:48:46.097921 9900 raft_consensus.cc:3037] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:46.097973 10106 raft_consensus.cc:3037] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:48:46.098644 9900 raft_consensus.cc:2454] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c93f1f1262444fb78d335310a66b7b45 in term 1.
I20260810 04:48:46.098661 10106 raft_consensus.cc:2454] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c93f1f1262444fb78d335310a66b7b45 in term 1.
I20260810 04:48:46.098811 9578 leader_election.cc:302] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c35cf4a5de4822badda84d6103959d, c93f1f1262444fb78d335310a66b7b45; no voters:
I20260810 04:48:46.098927 10220 raft_consensus.cc:2781] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:48:46.099125 10220 raft_consensus.cc:686] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 1 LEADER]: Becoming Leader. State: Replica: c93f1f1262444fb78d335310a66b7b45, State: Running, Role: LEADER
I20260810 04:48:46.099221 10220 consensus_queue.cc:237] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:48:46.100081 9456 catalog_manager.cc:5495] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 reported cstate change: term changed from 0 to 1, leader changed from <none> to c93f1f1262444fb78d335310a66b7b45 (127.14.237.1). New cstate: current_term: 1 leader_uuid: "c93f1f1262444fb78d335310a66b7b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } health_report { overall_health: UNKNOWN } } }
I20260810 04:48:46.112501 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:46.110494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.14.237.1:43207)
0810 04:48:46.110536 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:48:46.110537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:48:46.110544 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:48:46.110606 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:48:46.110606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:48:46.110622 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:48:46.110676 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:48:46.110773 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.111175 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.111881 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:48:46.112052 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:48:46.112102 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:48:46.112215 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:48:46.112302 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:48:46.112355 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:48:46.112389 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 04:48:46.112404 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:48:46.114138 10106 raft_consensus.cc:1261] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 1 FOLLOWER]: Refusing update from remote peer c93f1f1262444fb78d335310a66b7b45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:46.114180 9900 raft_consensus.cc:1261] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 1 FOLLOWER]: Refusing update from remote peer c93f1f1262444fb78d335310a66b7b45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:48:46.114473 10225 consensus_queue.cc:1035] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:46.114600 10220 consensus_queue.cc:1035] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:48:46.117517 10232 mvcc.cc:205] Tried to move back new op lower bound from 7316837687761223680 to 7316837687703285760. Current Snapshot: MvccSnapshot[applied={T|T < 7316837687761223680 or (T in {7316837687761223680})}]
I20260810 04:48:46.117841 10233 mvcc.cc:205] Tried to move back new op lower bound from 7316837687761223680 to 7316837687703285760. Current Snapshot: MvccSnapshot[applied={T|T < 7316837687761223680}]
I20260810 04:48:46.119091 9453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:48:46.119189 9453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:48:46.120061 9453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Sending DeleteTablet for 3 replicas of tablet 2c62c2ca861444159afba0509a14c16f
I20260810 04:48:46.120427 10078 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c62c2ca861444159afba0509a14c16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:46 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:48916
I20260810 04:48:46.120427 9659 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c62c2ca861444159afba0509a14c16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:46 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:56482
I20260810 04:48:46.120471 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 9548
I20260810 04:48:46.120642 9865 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c62c2ca861444159afba0509a14c16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:46 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:53952
I20260810 04:48:46.123296 10245 tablet_replica.cc:330] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: stopping tablet replica
I20260810 04:48:46.123377 10245 raft_consensus.cc:2229] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:48:46.123440 10245 raft_consensus.cc:2258] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:46.123576 10246 tablet_replica.cc:330] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: stopping tablet replica
I20260810 04:48:46.123658 10246 raft_consensus.cc:2229] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:48:46.123739 10246 raft_consensus.cc:2258] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:48:46.125411 10245 ts_tablet_manager.cc:1905] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:46.127087 10245 ts_tablet_manager.cc:1918] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:46.127151 10245 log.cc:1196] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/2c62c2ca861444159afba0509a14c16f
I20260810 04:48:46.127480 10245 ts_tablet_manager.cc:1939] T 2c62c2ca861444159afba0509a14c16f P 6aed075ee04640f3b0e6aceb2beed670: Deleting consensus metadata
I20260810 04:48:46.127982 9427 catalog_manager.cc:4841] TS 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865): tablet 2c62c2ca861444159afba0509a14c16f (table test-table [id=d3eebc41c4e645f885f184ed121b2da7]) successfully deleted
I20260810 04:48:46.129997 10246 ts_tablet_manager.cc:1905] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:48:46.131494 10246 ts_tablet_manager.cc:1918] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:48:46.131580 10246 log.cc:1196] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/2c62c2ca861444159afba0509a14c16f
I20260810 04:48:46.131875 10246 ts_tablet_manager.cc:1939] T 2c62c2ca861444159afba0509a14c16f P 06c35cf4a5de4822badda84d6103959d: Deleting consensus metadata
I20260810 04:48:46.132258 9431 catalog_manager.cc:4841] TS 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165): tablet 2c62c2ca861444159afba0509a14c16f (table test-table [id=d3eebc41c4e645f885f184ed121b2da7]) successfully deleted
W20260810 04:48:46.134716 9432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:43207 (error 108)
I20260810 04:48:46.135038 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 9771
I20260810 04:48:46.135213 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:48:46.135065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43207 (local address 127.0.0.1:56496)
0810 04:48:46.135131 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 04:48:46.135176 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:43207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 04:48:46.135325 9432 catalog_manager.cc:4584] TS c93f1f1262444fb78d335310a66b7b45 (127.14.237.1:43207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c62c2ca861444159afba0509a14c16f: Network error: Client connection negotiation failed: client connection to 127.14.237.1:43207: connect: Connection refused (error 111)
I20260810 04:48:46.146957 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 9979
I20260810 04:48:46.168394 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 9399
I20260810 04:49:06.178642 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38863
--webserver_interface=127.14.237.62
--webserver_port=39711
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:38863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:06.263886 10248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:06.264094 10248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:06.264117 10248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:06.265308 10248 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:49:06.265348 10248 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:49:06.265362 10248 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:49:06.265372 10248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:06.265385 10248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:06.265398 10248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:06.265410 10248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:06.265421 10248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:06.266831 10248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:38863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:06.267021 10248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:06.267189 10248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:06.276762 10260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:06.276772 10262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:06.276811 10248 server_base.cc:1034] running on GCE node
W20260810 04:49:06.276782 10258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:06.277377 10248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:06.277684 10248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:06.278856 10248 hybrid_clock.cc:648] HybridClock initialized: now 1786337346278845 us; error 37 us; skew 500 ppm
Aug 10 04:49:06 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337346, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:06.281478 10248 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:49:06.282016 10248 webserver.cc:460] Webserver started at http://127.14.237.62:39711/ using document root <none> and password file <none>
I20260810 04:49:06.282281 10248 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:06.282380 10248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:06.283933 10248 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:06.284649 10275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.284852 10248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:06.284957 10248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "5969816e2a4a4d6dae31219b44ed25e7"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b30ec9bef3ea3c9f31070bbf43495924"
server_key_iv: "5f5d685d14083f7beac467011b45afdc"
server_key_version: "encryptionkey@0"
I20260810 04:49:06.285456 10248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:06.326363 10248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:06.328770 10248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:06.328958 10248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:06.333467 10248 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:38863
I20260810 04:49:06.333518 10357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:38863 every 8 connection(s)
I20260810 04:49:06.334442 10248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:06.335326 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 10248
I20260810 04:49:06.335632 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:43207
--local_ip_for_outbound_sockets=127.14.237.1
--tserver_master_addrs=127.14.237.62:38863
--webserver_port=41591
--webserver_interface=127.14.237.1
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:49:06.336228 10358 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:49:06.336902 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Bootstrap starting.
I20260810 04:49:06.345750 10358 log.cc:824] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:06.347930 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:49:06.348276 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Bootstrap complete.
I20260810 04:49:06.350740 10358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:49:06.350983 10358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5969816e2a4a4d6dae31219b44ed25e7, State: Initialized, Role: FOLLOWER
I20260810 04:49:06.351227 10358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:49:06.351315 10358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:06.351372 10358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:06.351421 10358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:49:06.352589 10358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:49:06.352707 10358 leader_election.cc:302] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5969816e2a4a4d6dae31219b44ed25e7; no voters:
I20260810 04:49:06.352914 10358 leader_election.cc:288] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:49:06.352962 10363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:49:06.353149 10363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 2 LEADER]: Becoming Leader. State: Replica: 5969816e2a4a4d6dae31219b44ed25e7, State: Running, Role: LEADER
I20260810 04:49:06.353356 10363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:49:06.353225 10358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:06.353658 10369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5969816e2a4a4d6dae31219b44ed25e7. Latest consensus state: current_term: 2 leader_uuid: "5969816e2a4a4d6dae31219b44ed25e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } } }
I20260810 04:49:06.353885 10369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:06.353826 10368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5969816e2a4a4d6dae31219b44ed25e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } } }
I20260810 04:49:06.354169 10368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:06.354319 10381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:06.355198 10381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:06.355319 10381 catalog_manager.cc:1259] Loaded cluster ID: 3735365d8e0b4ec6beddde6322358dac
I20260810 04:49:06.355420 10381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:06.356554 10381 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:06.356742 10381 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Loaded TSK: 0
I20260810 04:49:06.379115 10381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Generated new TSK 1
I20260810 04:49:06.379243 10381 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:49:06.423267 10361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:06.423496 10361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:06.423539 10361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:06.424710 10361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:06.424775 10361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:06.424818 10361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:49:06.426347 10361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:43207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=41591
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:06.426672 10361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:06.426913 10361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:06.431998 10400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:06.432015 10401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:06.432112 10403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:06.432350 10361 server_base.cc:1034] running on GCE node
I20260810 04:49:06.432544 10361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:06.432796 10361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:06.433964 10361 hybrid_clock.cc:648] HybridClock initialized: now 1786337346433943 us; error 54 us; skew 500 ppm
Aug 10 04:49:06 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337346, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:06.437139 10361 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:49:06.437763 10361 webserver.cc:460] Webserver started at http://127.14.237.1:41591/ using document root <none> and password file <none>
I20260810 04:49:06.438056 10361 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:06.438145 10361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:06.439973 10361 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:49:06.440774 10418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.440958 10361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:06.441040 10361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "c93f1f1262444fb78d335310a66b7b45"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "547db94a2da0127c91c7ce06aa2df66b"
server_key_iv: "a2a3ed274ad3d086a66bde7af70e8bec"
server_key_version: "encryptionkey@0"
I20260810 04:49:06.441468 10361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:06.458168 10361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:06.460435 10361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:06.460594 10361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:06.460832 10361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:06.461294 10429 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:49:06.463198 10361 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:49:06.463260 10361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:49:06.463304 10361 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:49:06.463927 10361 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:49:06.463981 10361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:49:06.464051 10429 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Bootstrap starting.
I20260810 04:49:06.471344 10361 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:43207
I20260810 04:49:06.471412 10596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:43207 every 8 connection(s)
I20260810 04:49:06.471850 10361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:49:06.475451 10429 log.cc:824] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:06.478240 10429 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:49:06.478559 10429 tablet_bootstrap.cc:492] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Bootstrap complete.
I20260810 04:49:06.479803 10429 ts_tablet_manager.cc:1397] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
Aug 10 04:49:06 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337346, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:06.481452 10429 raft_consensus.cc:348] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:49:06.481791 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 10361
I20260810 04:49:06.481894 10429 raft_consensus.cc:729] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93f1f1262444fb78d335310a66b7b45, State: Initialized, Role: FOLLOWER
I20260810 04:49:06.482081 10429 consensus_queue.cc:260] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } } peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } }
I20260810 04:49:06.482124 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:40165
--local_ip_for_outbound_sockets=127.14.237.2
--tserver_master_addrs=127.14.237.62:38863
--webserver_port=45869
--webserver_interface=127.14.237.2
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:49:06.482725 10429 ts_tablet_manager.cc:1428] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260810 04:49:06.486140 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.472573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:40079)
0810 04:49:06.472765 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.472809 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:06.473330 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:06.476222 (+ 2892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:06.476226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:06.476557 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:06.476748 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.476754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.477991 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.477992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.478934 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.478937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.479543 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.479544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.479688 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.480278 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:06.480287 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:06.481800 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:06.484314 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:06.484321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:06.484323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:06.484550 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:06.484833 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:06.484834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:06.484834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:06.484942 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:06.485400 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:06.485408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:06.485574 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 04:49:06.485683 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 04:49:06.485869 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260810 04:49:06.486737 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.472689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:40079 (local address 127.14.237.62:38863)
0810 04:49:06.475717 (+ 3028us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.475727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.475770 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:49:06.475916 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.475924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.475989 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.476157 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:06.476885 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.477865 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.479027 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.479433 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.479561 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.481930 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:06.481935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:06.481936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:06.481949 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:06.484217 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.484640 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.484644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.484644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.484755 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.485008 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.485009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.485010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.485326 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:06.485421 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.486357 (+ 936us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.486510 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2965,"thread_start_us":95,"threads_started":1}
I20260810 04:49:06.487445 10598 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38863
I20260810 04:49:06.487546 10598 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:06.487725 10598 heartbeater.cc:511] Master 127.14.237.62:38863 requested a full tablet report, sending...
I20260810 04:49:06.488615 10305 ts_manager.cc:194] Registered new tserver with Master: c93f1f1262444fb78d335310a66b7b45 (127.14.237.1:43207)
I20260810 04:49:06.490101 10305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:40079
I20260810 04:49:06.491166 10598 heartbeater.cc:503] Master 127.14.237.62:38863 was elected leader, sending a full tablet report...
Aug 10 04:49:06 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337346, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.1@KRBTEST.COM
I20260810 04:49:06.499965 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.489506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43207 (local address 127.0.0.1:49240)
0810 04:49:06.489779 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.489810 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:06.489875 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:06.490101 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:06.490104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:06.490638 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:06.490723 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.490726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.492165 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.492166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.493249 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.493253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.493912 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.493914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.494041 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.494608 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:06.494614 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:06.496467 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:06.498731 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:06.498739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:06.498741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:06.499006 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:06.499228 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:06.499229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:06.499229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:06.499301 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:06.499592 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:06.499600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:06.499652 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 04:49:06.499740 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:49:06.499853 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":85,"threads_started":1}
I20260810 04:49:06.499985 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.489358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.14.237.1:43207)
0810 04:49:06.489719 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.489723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.489954 (+ 231us) server_negotiation.cc:407] Connection header received
0810 04:49:06.489985 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.489986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.489999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.490049 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:06.490809 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.492075 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.493341 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.493823 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.493898 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.496573 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:06.496583 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:06.496583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:06.496594 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:06.498617 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.499092 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.499095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.499095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.499160 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.499364 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.499365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.499365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.499529 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:06.499604 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.499782 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.499889 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":84,"threads_started":1}
I20260810 04:49:06.501052 10493 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c62c2ca861444159afba0509a14c16f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:48:46 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:49240
I20260810 04:49:06.501669 10614 tablet_replica.cc:330] stopping tablet replica
I20260810 04:49:06.501824 10614 raft_consensus.cc:2229] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:06.501935 10614 raft_consensus.cc:2258] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:06.502372 10614 ts_tablet_manager.cc:1905] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:06.504778 10614 ts_tablet_manager.cc:1918] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:06.504923 10614 log.cc:1196] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal/wals/2c62c2ca861444159afba0509a14c16f
I20260810 04:49:06.505327 10614 ts_tablet_manager.cc:1939] T 2c62c2ca861444159afba0509a14c16f P c93f1f1262444fb78d335310a66b7b45: Deleting consensus metadata
I20260810 04:49:06.506403 10279 catalog_manager.cc:4841] TS c93f1f1262444fb78d335310a66b7b45 (127.14.237.1:43207): tablet 2c62c2ca861444159afba0509a14c16f (table test-table [id=d3eebc41c4e645f885f184ed121b2da7]) successfully deleted
W20260810 04:49:06.574671 10608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:06.574908 10608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:06.574949 10608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:06.576120 10608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:06.576184 10608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:06.576227 10608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:49:06.577773 10608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:40165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=45869
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:06.578032 10608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:06.578274 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:06.583467 10624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:06.583475 10626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:06.583549 10628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:06.583635 10608 server_base.cc:1034] running on GCE node
I20260810 04:49:06.583895 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:06.584169 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:06.585335 10608 hybrid_clock.cc:648] HybridClock initialized: now 1786337346585322 us; error 44 us; skew 500 ppm
Aug 10 04:49:06 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337346, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:06.587869 10608 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:49:06.588282 10608 webserver.cc:460] Webserver started at http://127.14.237.2:45869/ using document root <none> and password file <none>
I20260810 04:49:06.588557 10608 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:06.588644 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:06.590067 10608 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:06.590636 10639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.590799 10608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:06.590894 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "06c35cf4a5de4822badda84d6103959d"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "8064110840cb3532bee667e3467c8f9d"
server_key_iv: "486f0914bb9848e287c863f3f6e86704"
server_key_version: "encryptionkey@0"
I20260810 04:49:06.591238 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:06.612363 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:06.614370 10608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:06.614523 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:06.614779 10608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:06.615088 10608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:06.615140 10608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.615181 10608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:06.615213 10608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.620957 10608 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:40165
I20260810 04:49:06.621027 10819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:40165 every 8 connection(s)
I20260810 04:49:06.621423 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:49:06 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337346, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:06.629642 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 10608
I20260810 04:49:06.629966 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:43865
--local_ip_for_outbound_sockets=127.14.237.3
--tserver_master_addrs=127.14.237.62:38863
--webserver_port=34393
--webserver_interface=127.14.237.3
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:49:06.630931 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.622165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:38931)
0810 04:49:06.622368 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.622403 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:06.622885 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:06.623109 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:06.623114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:06.623705 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:06.623889 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.623894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.624591 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.624592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.625499 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.625502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.625863 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.625865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.625971 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.626498 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:06.626507 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:06.627951 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:06.629513 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:06.629520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:06.629521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:06.629818 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:06.630050 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:06.630051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:06.630051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:06.630208 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:06.630469 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:06.630474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:06.630579 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 04:49:06.630666 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:49:06.630777 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
I20260810 04:49:06.630946 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.622167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:38931 (local address 127.14.237.62:38863)
0810 04:49:06.622231 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.622234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.622486 (+ 252us) server_negotiation.cc:407] Connection header received
0810 04:49:06.622964 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.622966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.622980 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.623034 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:06.623963 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.624489 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.625578 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.625773 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.625824 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.628032 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:06.628033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:06.628034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:06.628040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:06.629429 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.629888 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.629890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.629890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.629966 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.630259 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.630260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.630260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.630403 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:06.630474 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.630709 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.630815 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:49:06.631553 10820 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38863
I20260810 04:49:06.631660 10820 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:06.631853 10820 heartbeater.cc:511] Master 127.14.237.62:38863 requested a full tablet report, sending...
I20260810 04:49:06.632299 10305 ts_manager.cc:194] Registered new tserver with Master: 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165)
I20260810 04:49:06.632678 10305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:38931
W20260810 04:49:06.714325 10829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:06.714522 10829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:06.714545 10829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:06.715700 10829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:06.715770 10829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:06.715811 10829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:49:06.717288 10829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:43865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=34393
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:06.717552 10829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:06.717787 10829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:06.722795 10837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:06.722827 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:06.722827 10838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:06.723153 10829 server_base.cc:1034] running on GCE node
I20260810 04:49:06.723388 10829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:06.723644 10829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:06.724957 10829 hybrid_clock.cc:648] HybridClock initialized: now 1786337346724923 us; error 40 us; skew 500 ppm
Aug 10 04:49:06 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337346, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:06.727559 10829 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:49:06.728034 10829 webserver.cc:460] Webserver started at http://127.14.237.3:34393/ using document root <none> and password file <none>
I20260810 04:49:06.728271 10829 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:06.728354 10829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:06.729768 10829 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:49:06.730330 10853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.730521 10829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:06.730620 10829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "6aed075ee04640f3b0e6aceb2beed670"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b50d2258b9c8262703dd14f4a476a8af"
server_key_iv: "28fb30499ce80210b944651ae06180ff"
server_key_version: "encryptionkey@0"
I20260810 04:49:06.730966 10829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:06.769713 10829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:06.771736 10829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:06.771930 10829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:06.772215 10829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:06.772549 10829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:06.772603 10829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.772639 10829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:06.772679 10829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.778688 10829 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:43865
I20260810 04:49:06.778743 11032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:43865 every 8 connection(s)
I20260810 04:49:06.779085 10829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:49:06 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337346, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:06.786804 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 10829
I20260810 04:49:06.789386 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.779836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:55687 (local address 127.14.237.62:38863)
0810 04:49:06.779910 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.779912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.780139 (+ 227us) server_negotiation.cc:407] Connection header received
0810 04:49:06.780651 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.780653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.780664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.780728 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:06.781769 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.782260 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.783262 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.783447 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.783499 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.785790 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:06.785793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:06.785793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:06.785801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:06.787803 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.788229 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.788231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.788231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.788294 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.788627 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.788628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.788628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.788765 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:06.788835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.789142 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.789288 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:49:06.789510 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.779812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:55687)
0810 04:49:06.780011 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.780057 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:06.780563 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:06.780797 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:06.780801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:06.781469 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:06.781685 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.781691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.782345 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.782345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.783173 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.783175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.783521 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.783522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.783643 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.784137 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:06.784147 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:06.785647 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:06.787891 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:06.787898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:06.787900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:06.788131 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:06.788365 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:06.788366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:06.788366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:06.788552 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:06.788832 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:06.788836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:06.788991 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 04:49:06.789101 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 04:49:06.789300 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260810 04:49:06.790059 11034 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38863
I20260810 04:49:06.790163 11034 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:06.790350 11034 heartbeater.cc:511] Master 127.14.237.62:38863 requested a full tablet report, sending...
I20260810 04:49:06.790776 10305 ts_manager.cc:194] Registered new tserver with Master: 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865)
I20260810 04:49:06.791167 10305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:55687
I20260810 04:49:06.794245 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.787268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.14.237.62:38863)
0810 04:49:06.787407 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.787409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.787484 (+ 75us) server_negotiation.cc:407] Connection header received
0810 04:49:06.787533 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.787534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.787548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.787646 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:06.788286 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.788954 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.790065 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.790355 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.790431 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.791768 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:06.791770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:06.791771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:06.791778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:06.793014 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.793431 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.793433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.793433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.793494 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.793697 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.793698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.793698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.793807 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:06.794010 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.794050 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.794129 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260810 04:49:06.794976 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:49:06.797464 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.795333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.14.237.62:38863)
0810 04:49:06.795379 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.795381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.795529 (+ 148us) server_negotiation.cc:407] Connection header received
0810 04:49:06.795574 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.795575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.795603 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.795664 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:49:06.795738 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.796159 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.796849 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.797024 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.797073 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.797259 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:49:06.797310 (+ 51us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 04:49:06.797425 (+ 115us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53758: token expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260810 04:49:06.797525 11047 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53758: token expired
W20260810 04:49:06.798511 10192 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.237.62:38863)
I20260810 04:49:06.804028 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.798763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.14.237.62:38863)
0810 04:49:06.798816 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.798818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.798826 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:49:06.798867 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.798867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.798875 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.798917 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:06.799345 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.799769 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.800397 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.800562 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.800625 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.801706 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:06.801707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:06.801708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:06.801714 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:06.803123 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.803413 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.803413 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.803413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.803465 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:06.803632 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:06.803632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:06.803632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:06.803719 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:06.803788 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.803875 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.803932 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:49:06.805135 10305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:49:06.806833 10305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:49:06.809046 10493 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10 (DEFAULT_TABLE table=test-table [id=d3718c21e27049dc9383ac9ee9fc9b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:06.809288 10493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:06.810730 10429 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Bootstrap starting.
I20260810 04:49:06.811048 10429 tablet_bootstrap.cc:654] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:06.811854 10429 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: No bootstrap required, opened a new log
I20260810 04:49:06.811928 10429 ts_tablet_manager.cc:1397] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:06.812091 10429 raft_consensus.cc:348] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.812171 10429 raft_consensus.cc:374] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:06.812229 10429 raft_consensus.cc:729] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93f1f1262444fb78d335310a66b7b45, State: Initialized, Role: FOLLOWER
I20260810 04:49:06.812276 10429 consensus_queue.cc:260] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.812376 10429 ts_tablet_manager.cc:1428] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:06.813342 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.808754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:43865 (local address 127.0.0.1:53636)
0810 04:49:06.808786 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.808798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:06.808840 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:06.809489 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:06.809490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:06.809493 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:06.809540 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.809541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.810877 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.810878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.812084 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.812087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.813053 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.813053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.813126 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.813144 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 04:49:06.813191 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 04:49:06.813276 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 04:49:06.813405 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.809059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.14.237.3:43865)
0810 04:49:06.809318 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.809323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.809335 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:49:06.809377 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.809379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.809395 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.809470 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:06.809633 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.810776 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.812196 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.812938 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.813045 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.813095 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.813275 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.813321 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260810 04:49:06.813519 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.809051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.14.237.2:40165)
0810 04:49:06.809618 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.809624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.809634 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:49:06.809674 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.809676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.809690 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.809770 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:06.809989 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.811010 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.812213 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.812938 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.813014 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.813078 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.813381 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.813430 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":90,"threads_started":1}
I20260810 04:49:06.813776 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.808828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40165 (local address 127.0.0.1:44652)
0810 04:49:06.809256 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.809269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:06.809326 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:06.809869 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:06.809871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:06.809873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:06.809912 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.809913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.811093 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.811094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.812124 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.812126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.813115 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.813116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.813202 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.813207 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:49:06.813279 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:49:06.813321 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":59,"threads_started":1}
I20260810 04:49:06.814154 10930 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10 (DEFAULT_TABLE table=test-table [id=d3718c21e27049dc9383ac9ee9fc9b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:06.814494 10930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:06.817246 11057 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: Bootstrap starting.
I20260810 04:49:06.818015 11057 tablet_bootstrap.cc:654] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:06.817895 10716 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10 (DEFAULT_TABLE table=test-table [id=d3718c21e27049dc9383ac9ee9fc9b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:06.818298 10716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:06.818658 11057 log.cc:824] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:06.819492 11057 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: No bootstrap required, opened a new log
I20260810 04:49:06.819569 11057 ts_tablet_manager.cc:1397] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:06.821669 11057 raft_consensus.cc:348] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.821867 11057 raft_consensus.cc:374] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:06.821916 11057 raft_consensus.cc:729] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aed075ee04640f3b0e6aceb2beed670, State: Initialized, Role: FOLLOWER
I20260810 04:49:06.822012 11059 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: Bootstrap starting.
I20260810 04:49:06.822044 11057 consensus_queue.cc:260] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.822327 11057 ts_tablet_manager.cc:1428] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 04:49:06.822371 11034 heartbeater.cc:503] Master 127.14.237.62:38863 was elected leader, sending a full tablet report...
I20260810 04:49:06.822715 11059 tablet_bootstrap.cc:654] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:06.823342 11059 log.cc:824] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:06.824023 11059 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: No bootstrap required, opened a new log
I20260810 04:49:06.824077 11059 ts_tablet_manager.cc:1397] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:06.825521 11059 raft_consensus.cc:348] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.825711 11059 raft_consensus.cc:374] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:06.825748 11059 raft_consensus.cc:729] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c35cf4a5de4822badda84d6103959d, State: Initialized, Role: FOLLOWER
I20260810 04:49:06.825853 11059 consensus_queue.cc:260] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.826071 11059 ts_tablet_manager.cc:1428] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:06.826126 10820 heartbeater.cc:503] Master 127.14.237.62:38863 was elected leader, sending a full tablet report...
I20260810 04:49:06.841500 11060 raft_consensus.cc:482] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:06.841590 11060 raft_consensus.cc:504] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.841924 11060 leader_election.cc:288] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165), c93f1f1262444fb78d335310a66b7b45 (127.14.237.1:43207)
I20260810 04:49:06.844956 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.842107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40165 (local address 127.14.237.3:50607)
0810 04:49:06.842242 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.842252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:06.842314 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:06.842444 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:06.842445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:06.842446 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:06.842539 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.842541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.843110 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.843111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.844141 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.844144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.844687 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.844688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.844767 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.844772 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:49:06.844828 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:49:06.844867 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260810 04:49:06.844959 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.842128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:50607 (local address 127.14.237.2:40165)
0810 04:49:06.842174 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.842177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.842321 (+ 144us) server_negotiation.cc:407] Connection header received
0810 04:49:06.842356 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.842357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.842364 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.842430 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:06.842617 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.843035 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.844230 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.844613 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.844669 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.844675 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.844848 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.844885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:49:06.845192 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.842061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43207 (local address 127.14.237.3:53691)
0810 04:49:06.842171 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.842183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:06.842241 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:06.842400 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:06.842401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:06.842403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:06.842475 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.842477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.843152 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.843153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.844154 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:06.844157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.844862 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.844863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:06.844983 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.844987 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:06.845040 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:49:06.845077 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":68}
I20260810 04:49:06.845290 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.842147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:53691 (local address 127.14.237.1:43207)
0810 04:49:06.842193 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.842195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.842242 (+ 47us) server_negotiation.cc:407] Connection header received
0810 04:49:06.842297 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.842298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.842326 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.842390 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:06.842544 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.843039 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.844286 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.844761 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.844848 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.844911 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.845098 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.845170 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:49:06.845325 10746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6ae54a56c944a6b7ce01c6f3b9ab10" candidate_uuid: "6aed075ee04640f3b0e6aceb2beed670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c35cf4a5de4822badda84d6103959d" is_pre_election: true
I20260810 04:49:06.845441 10746 raft_consensus.cc:2454] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aed075ee04640f3b0e6aceb2beed670 in term 0.
I20260810 04:49:06.845666 10855 leader_election.cc:302] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06c35cf4a5de4822badda84d6103959d, 6aed075ee04640f3b0e6aceb2beed670; no voters:
I20260810 04:49:06.845654 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6ae54a56c944a6b7ce01c6f3b9ab10" candidate_uuid: "6aed075ee04640f3b0e6aceb2beed670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93f1f1262444fb78d335310a66b7b45" is_pre_election: true
I20260810 04:49:06.845793 10529 raft_consensus.cc:2454] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aed075ee04640f3b0e6aceb2beed670 in term 0.
I20260810 04:49:06.845908 11060 raft_consensus.cc:2781] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:49:06.845985 11060 raft_consensus.cc:482] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:49:06.846015 11060 raft_consensus.cc:3037] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:06.846719 11060 raft_consensus.cc:504] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.846863 11060 leader_election.cc:288] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [CANDIDATE]: Term 1 election: Requested vote from peers 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165), c93f1f1262444fb78d335310a66b7b45 (127.14.237.1:43207)
I20260810 04:49:06.847023 10746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6ae54a56c944a6b7ce01c6f3b9ab10" candidate_uuid: "6aed075ee04640f3b0e6aceb2beed670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c35cf4a5de4822badda84d6103959d"
I20260810 04:49:06.847054 10529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6ae54a56c944a6b7ce01c6f3b9ab10" candidate_uuid: "6aed075ee04640f3b0e6aceb2beed670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93f1f1262444fb78d335310a66b7b45"
I20260810 04:49:06.847124 10529 raft_consensus.cc:3037] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:06.847127 10746 raft_consensus.cc:3037] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:06.847817 10529 raft_consensus.cc:2454] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aed075ee04640f3b0e6aceb2beed670 in term 1.
I20260810 04:49:06.847870 10746 raft_consensus.cc:2454] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aed075ee04640f3b0e6aceb2beed670 in term 1.
I20260810 04:49:06.848016 10856 leader_election.cc:302] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aed075ee04640f3b0e6aceb2beed670, c93f1f1262444fb78d335310a66b7b45; no voters:
I20260810 04:49:06.848114 11060 raft_consensus.cc:2781] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:06.848242 11060 raft_consensus.cc:686] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 1 LEADER]: Becoming Leader. State: Replica: 6aed075ee04640f3b0e6aceb2beed670, State: Running, Role: LEADER
I20260810 04:49:06.848335 11060 consensus_queue.cc:237] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:06.849112 10303 catalog_manager.cc:5495] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3). New cstate: current_term: 1 leader_uuid: "6aed075ee04640f3b0e6aceb2beed670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260810 04:49:06.859256 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.857270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.14.237.3:43865)
0810 04:49:06.857309 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:06.857311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:06.857320 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:49:06.857358 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:06.857359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:06.857372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:06.857428 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:49:06.857518 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.857936 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.858595 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:06.858775 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:06.858824 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:06.858956 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:49:06.859039 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:49:06.859096 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:06.859144 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 04:49:06.859158 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:49:06.860836 10746 raft_consensus.cc:1261] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 1 FOLLOWER]: Refusing update from remote peer 6aed075ee04640f3b0e6aceb2beed670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:06.860836 10529 raft_consensus.cc:1261] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Refusing update from remote peer 6aed075ee04640f3b0e6aceb2beed670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:06.861186 11064 consensus_queue.cc:1035] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:06.861317 11060 consensus_queue.cc:1035] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:06.864527 11084 mvcc.cc:205] Tried to move back new op lower bound from 7316837772739747840 to 7316837772691410944. Current Snapshot: MvccSnapshot[applied={T|T < 7316837772739747840}]
I20260810 04:49:06.865402 10303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:06.865500 10303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:06.866272 10303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Sending DeleteTablet for 3 replicas of tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10
I20260810 04:49:06.866573 10930 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:06 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:53636
I20260810 04:49:06.866601 10716 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:06 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:44652
I20260810 04:49:06.866761 10493 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:06 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:49240
I20260810 04:49:06.866860 11096 tablet_replica.cc:330] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: stopping tablet replica
I20260810 04:49:06.866888 10614 tablet_replica.cc:330] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: stopping tablet replica
I20260810 04:49:06.866933 10614 raft_consensus.cc:2229] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:06.866994 11096 raft_consensus.cc:2229] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:06.867017 10614 raft_consensus.cc:2258] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:06.867100 11096 raft_consensus.cc:2258] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:06.867297 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 10361
I20260810 04:49:06.881318 11094 tablet_replica.cc:330] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: stopping tablet replica
I20260810 04:49:06.881448 11094 raft_consensus.cc:2229] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:49:06.881553 11094 raft_consensus.cc:2258] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:06.883268 11094 ts_tablet_manager.cc:1905] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:06.884087 11096 ts_tablet_manager.cc:1905] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:06.885239 11094 ts_tablet_manager.cc:1918] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:06.885317 11094 log.cc:1196] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/4a6ae54a56c944a6b7ce01c6f3b9ab10
I20260810 04:49:06.885650 11094 ts_tablet_manager.cc:1939] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 6aed075ee04640f3b0e6aceb2beed670: Deleting consensus metadata
I20260810 04:49:06.886087 10277 catalog_manager.cc:4841] TS 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865): tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10 (table test-table [id=d3718c21e27049dc9383ac9ee9fc9b53]) successfully deleted
I20260810 04:49:06.886495 11096 ts_tablet_manager.cc:1918] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:06.886585 11096 log.cc:1196] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/4a6ae54a56c944a6b7ce01c6f3b9ab10
I20260810 04:49:06.886955 11096 ts_tablet_manager.cc:1939] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P 06c35cf4a5de4822badda84d6103959d: Deleting consensus metadata
I20260810 04:49:06.902930 10278 catalog_manager.cc:4841] TS 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165): tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10 (table test-table [id=d3718c21e27049dc9383ac9ee9fc9b53]) successfully deleted
W20260810 04:49:06.908430 10279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:43207 (error 108)
I20260810 04:49:06.908756 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 10608
I20260810 04:49:06.910538 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:06.908876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:43207 (local address 127.0.0.1:49256)
0810 04:49:06.910413 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:06.910494 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:43207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1516}
W20260810 04:49:06.910681 10279 catalog_manager.cc:4584] TS c93f1f1262444fb78d335310a66b7b45 (127.14.237.1:43207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a6ae54a56c944a6b7ce01c6f3b9ab10: Network error: Client connection negotiation failed: client connection to 127.14.237.1:43207: connect: Connection refused (error 111)
I20260810 04:49:06.917687 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 10829
I20260810 04:49:06.930789 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 10248
I20260810 04:49:26.940663 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38863
--webserver_interface=127.14.237.62
--webserver_port=39711
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:38863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:27.021829 11103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:27.022025 11103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:27.022046 11103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:27.023056 11103 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:49:27.023106 11103 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:49:27.023120 11103 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:49:27.023131 11103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:27.023144 11103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:27.023157 11103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:27.023169 11103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:27.023181 11103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:27.024528 11103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:38863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:27.024705 11103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:27.024879 11103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:27.029732 11115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:27.029767 11110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:27.029906 11103 server_base.cc:1034] running on GCE node
W20260810 04:49:27.029737 11111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:27.030247 11103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:27.030539 11103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:27.031724 11103 hybrid_clock.cc:648] HybridClock initialized: now 1786337367031717 us; error 37 us; skew 500 ppm
Aug 10 04:49:27 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337367, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:27.034185 11103 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:49:27.034636 11103 webserver.cc:460] Webserver started at http://127.14.237.62:39711/ using document root <none> and password file <none>
I20260810 04:49:27.034853 11103 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:27.034920 11103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:27.036238 11103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:27.036890 11126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.037061 11103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:27.037173 11103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "5969816e2a4a4d6dae31219b44ed25e7"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b30ec9bef3ea3c9f31070bbf43495924"
server_key_iv: "5f5d685d14083f7beac467011b45afdc"
server_key_version: "encryptionkey@0"
I20260810 04:49:27.037544 11103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:27.051448 11103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:27.053329 11103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:27.053493 11103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:27.057658 11103 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:38863
I20260810 04:49:27.057693 11205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:38863 every 8 connection(s)
I20260810 04:49:27.058063 11103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:27.059481 11208 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:49:27.060122 11208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Bootstrap starting.
I20260810 04:49:27.066334 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 11103
I20260810 04:49:27.066619 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:43207
--local_ip_for_outbound_sockets=127.14.237.1
--tserver_master_addrs=127.14.237.62:38863
--webserver_port=41591
--webserver_interface=127.14.237.1
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:49:27.069295 11208 log.cc:824] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:27.071600 11208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:49:27.071938 11208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Bootstrap complete.
I20260810 04:49:27.074494 11208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:49:27.074755 11208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5969816e2a4a4d6dae31219b44ed25e7, State: Initialized, Role: FOLLOWER
I20260810 04:49:27.074973 11208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:49:27.075090 11208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:27.075147 11208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:27.075198 11208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 2 FOLLOWER]: Advancing to term 3
I20260810 04:49:27.076208 11208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:49:27.076339 11208 leader_election.cc:302] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5969816e2a4a4d6dae31219b44ed25e7; no voters:
I20260810 04:49:27.076529 11208 leader_election.cc:288] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 04:49:27.076615 11214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 04:49:27.076807 11208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:27.076834 11214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [term 3 LEADER]: Becoming Leader. State: Replica: 5969816e2a4a4d6dae31219b44ed25e7, State: Running, Role: LEADER
I20260810 04:49:27.076947 11214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } }
I20260810 04:49:27.077319 11216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5969816e2a4a4d6dae31219b44ed25e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } } }
I20260810 04:49:27.077425 11216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:27.077605 11216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5969816e2a4a4d6dae31219b44ed25e7. Latest consensus state: current_term: 3 leader_uuid: "5969816e2a4a4d6dae31219b44ed25e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5969816e2a4a4d6dae31219b44ed25e7" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38863 } } }
I20260810 04:49:27.077687 11216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:27.078078 11227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:27.078991 11227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:27.079111 11227 catalog_manager.cc:1259] Loaded cluster ID: 3735365d8e0b4ec6beddde6322358dac
I20260810 04:49:27.079149 11227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:27.080554 11227 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:27.080761 11227 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Loaded TSK: 0
I20260810 04:49:27.080806 11227 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5969816e2a4a4d6dae31219b44ed25e7: Loaded TSK: 1
F20260810 04:49:27.081079 11227 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbf39d0e9ad google::LogMessage::Fail() at ??:0
@ 0x7fbf39d1188a google::LogMessage::SendToLog() at ??:0
@ 0x7fbf39d0e4de google::LogMessage::Flush() at ??:0
@ 0x7fbf39d0fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbf3c47b104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbf3c478d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbf41c681cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbf41c6848e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbf41c68594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbf41c2ec48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fbf41c38ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbf3bbdc0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbf3bbd1954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbf3ca356db start_thread at ??:0
@ 0x7fbf3880c61f clone at ??:0
W20260810 04:49:27.171115 11212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:27.171343 11212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:27.171381 11212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:27.172591 11212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:27.172654 11212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:27.172710 11212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:49:27.174290 11212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:43207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=41591
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:27.174548 11212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:27.174786 11212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:27.180655 11269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:27.180665 11272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:27.180775 11268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:27.181133 11212 server_base.cc:1034] running on GCE node
I20260810 04:49:27.181370 11212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:27.181640 11212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:27.182801 11212 hybrid_clock.cc:648] HybridClock initialized: now 1786337367182781 us; error 35 us; skew 500 ppm
Aug 10 04:49:27 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337367, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:27.185632 11212 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:49:27.186168 11212 webserver.cc:460] Webserver started at http://127.14.237.1:41591/ using document root <none> and password file <none>
I20260810 04:49:27.186445 11212 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:27.186528 11212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:27.188354 11212 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:49:27.189168 11286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.189399 11212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:27.189508 11212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "c93f1f1262444fb78d335310a66b7b45"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "547db94a2da0127c91c7ce06aa2df66b"
server_key_iv: "a2a3ed274ad3d086a66bde7af70e8bec"
server_key_version: "encryptionkey@0"
I20260810 04:49:27.189968 11212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:27.220618 11212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:27.223763 11212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:27.223949 11212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:27.224208 11212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:27.224694 11296 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:49:27.226851 11212 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:49:27.226935 11212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:49:27.226995 11212 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:49:27.227723 11212 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:49:27.227789 11212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:49:27.227865 11296 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Bootstrap starting.
I20260810 04:49:27.235386 11212 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:43207
I20260810 04:49:27.235520 11485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:43207 every 8 connection(s)
I20260810 04:49:27.235819 11212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:49:27.239138 11296 log.cc:824] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:27.242115 11296 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:49:27.242491 11296 tablet_bootstrap.cc:492] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Bootstrap complete.
I20260810 04:49:27.243325 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 11212
I20260810 04:49:27.243608 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:40165
--local_ip_for_outbound_sockets=127.14.237.2
--tserver_master_addrs=127.14.237.62:38863
--webserver_port=45869
--webserver_interface=127.14.237.2
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:49:27.243710 11296 ts_tablet_manager.cc:1397] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260810 04:49:27.245308 11296 raft_consensus.cc:348] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:27.245682 11296 raft_consensus.cc:729] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93f1f1262444fb78d335310a66b7b45, State: Initialized, Role: FOLLOWER
I20260810 04:49:27.245832 11296 consensus_queue.cc:260] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:27.246126 11296 ts_tablet_manager.cc:1428] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260810 04:49:27.311398 11494 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:49:27.236558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:55309)
0810 04:49:27.236741 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.236784 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:27.237427 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:27.311165 (+ 73738us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
W20260810 04:49:27.311710 11490 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:38863 (0 consecutive failures): Network error: Failed to ping master at 127.14.237.62:38863: Client connection negotiation failed: client connection to 127.14.237.62:38863: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:49:27.312100 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:27.311960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:34223)
0810 04:49:27.312021 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.312067 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:49:27.312600 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:27.312495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:35935)
0810 04:49:27.312555 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.312583 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 04:49:27.312762 11490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:49:27.338285 11496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:27.338512 11496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:27.338551 11496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:27.339689 11496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:27.339752 11496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:27.339797 11496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:49:27.341327 11496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:40165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=45869
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:27.341598 11496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:27.341827 11496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:27.346941 11512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:27.346935 11505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:27.347273 11496 server_base.cc:1034] running on GCE node
W20260810 04:49:27.346938 11504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:27.347564 11496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:27.347819 11496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:27.348985 11496 hybrid_clock.cc:648] HybridClock initialized: now 1786337367348965 us; error 37 us; skew 500 ppm
Aug 10 04:49:27 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337367, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:27.351485 11496 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:49:27.351956 11496 webserver.cc:460] Webserver started at http://127.14.237.2:45869/ using document root <none> and password file <none>
I20260810 04:49:27.352140 11496 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:27.352180 11496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:27.353591 11496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:27.354187 11522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.354383 11496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:27.354477 11496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "06c35cf4a5de4822badda84d6103959d"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "8064110840cb3532bee667e3467c8f9d"
server_key_iv: "486f0914bb9848e287c863f3f6e86704"
server_key_version: "encryptionkey@0"
I20260810 04:49:27.354835 11496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:27.384882 11496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:27.387125 11496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:27.387300 11496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:27.387555 11496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:27.387887 11496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:27.387943 11496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.387988 11496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:27.388012 11496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.394328 11496 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:40165
I20260810 04:49:27.394441 11695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:40165 every 8 connection(s)
I20260810 04:49:27.394737 11496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:49:27.395748 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:27.395378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:56871)
0810 04:49:27.395532 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.395601 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
W20260810 04:49:27.396152 11697 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:38863 (0 consecutive failures): Network error: Failed to ping master at 127.14.237.62:38863: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
I20260810 04:49:27.396425 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:27.396337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:46345)
0810 04:49:27.396382 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.396407 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 04:49:27.396665 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:27.396618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:43021)
0810 04:49:27.396638 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.396655 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260810 04:49:27.396776 11697 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:49:27.397581 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 11496
I20260810 04:49:27.397862 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:43865
--local_ip_for_outbound_sockets=127.14.237.3
--tserver_master_addrs=127.14.237.62:38863
--webserver_port=34393
--webserver_interface=127.14.237.3
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:27.480592 11700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:27.480842 11700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:27.480883 11700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:27.482028 11700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:27.482112 11700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:27.482157 11700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:49:27.483624 11700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:43865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=34393
--tserver_master_addrs=127.14.237.62:38863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:27.483893 11700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:27.484128 11700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:27.489027 11715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:27.489003 11710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:27.489001 11712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:27.489096 11700 server_base.cc:1034] running on GCE node
I20260810 04:49:27.489495 11700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:27.489759 11700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:27.490942 11700 hybrid_clock.cc:648] HybridClock initialized: now 1786337367490929 us; error 24 us; skew 500 ppm
Aug 10 04:49:27 dist-test-slave-8hhm krb5kdc[9378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337367, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:27.493747 11700 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:49:27.494275 11700 webserver.cc:460] Webserver started at http://127.14.237.3:34393/ using document root <none> and password file <none>
I20260810 04:49:27.494590 11700 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:27.494668 11700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:27.495966 11700 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:27.496558 11725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.496791 11700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.496909 11700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "6aed075ee04640f3b0e6aceb2beed670"
format_stamp: "Formatted at 2026-08-10 04:48:45 on dist-test-slave-8hhm"
server_key: "b50d2258b9c8262703dd14f4a476a8af"
server_key_iv: "28fb30499ce80210b944651ae06180ff"
server_key_version: "encryptionkey@0"
I20260810 04:49:27.497313 11700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:27.509532 11700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:27.511482 11700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:27.511657 11700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:27.511895 11700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:27.512216 11700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:27.512275 11700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.512319 11700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:27.512364 11700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:27.519196 11700 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:43865
I20260810 04:49:27.519232 11896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:43865 every 8 connection(s)
I20260810 04:49:27.519582 11700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:49:27.520794 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:27.520443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:38227)
0810 04:49:27.520595 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.520662 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20260810 04:49:27.521309 11898 heartbeater.cc:644] Failed to heartbeat to 127.14.237.62:38863 (0 consecutive failures): Network error: Failed to ping master at 127.14.237.62:38863: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
I20260810 04:49:27.521626 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:27.521506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:40083)
0810 04:49:27.521561 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.521604 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:49:27.521983 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:27.521867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:56857)
0810 04:49:27.521929 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:27.521960 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 04:49:27.522159 11898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:49:27.523648 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 11700
I20260810 04:49:28.313452 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:28.313086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:43493)
0810 04:49:28.313202 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:28.313387 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260810 04:49:28.397390 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:28.397109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:40079)
0810 04:49:28.397312 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:28.397345 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260810 04:49:28.446429 11910 raft_consensus.cc:482] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:28.446554 11910 raft_consensus.cc:504] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:28.446846 11910 leader_election.cc:288] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865), 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165)
Aug 10 04:49:28 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337367, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.3@KRBTEST.COM
Aug 10 04:49:28 dist-test-slave-8hhm krb5kdc[9378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337367, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.2@KRBTEST.COM
I20260810 04:49:28.457777 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:28.447161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:37447 (local address 127.14.237.2:40165)
0810 04:49:28.447295 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:28.447309 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:28.447347 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:49:28.447914 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:28.447916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:28.447975 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:28.448138 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:28.448901 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:28.449823 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:28.450870 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:28.451290 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:28.451402 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:28.454272 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:28.454276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:28.454276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:28.454288 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:28.456031 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:28.456441 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:28.456444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:28.456445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:28.456548 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:28.456820 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:28.456821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:28.456822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:28.457070 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:28.457133 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:28.457511 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 04:49:28.457666 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260810 04:49:28.457820 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:28.447005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:40165 (local address 127.14.237.1:37447)
0810 04:49:28.447075 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:28.447085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:28.447138 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:28.448194 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:28.448196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:28.448559 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:28.448771 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:28.448777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:28.449941 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:28.449942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:28.450770 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:28.450772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:28.451389 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:28.451390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:28.451497 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:28.452241 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:28.452245 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:28.454160 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:28.456115 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:28.456116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:28.456116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:28.456359 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:28.456629 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:28.456629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:28.456630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:28.456748 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:28.457148 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:28.457155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:28.457317 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 04:49:28.457447 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 04:49:28.457695 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":150}
I20260810 04:49:28.458046 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:28.447192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:36025 (local address 127.14.237.3:43865)
0810 04:49:28.447356 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:28.447365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:28.447383 (+ 18us) server_negotiation.cc:407] Connection header received
0810 04:49:28.447902 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:28.447905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:28.447975 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:28.448170 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:28.448898 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:28.449843 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:28.450925 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:28.451307 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:28.451440 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:28.453876 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:28.453879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:28.453880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:28.453890 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:28.455954 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:28.456466 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:28.456469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:28.456469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:28.456546 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:28.456826 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:28.456844 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:28.456845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:28.457070 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:28.457196 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:28.457807 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 04:49:28.457938 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260810 04:49:28.458416 11916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:28.447019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:43865 (local address 127.14.237.1:36025)
0810 04:49:28.447223 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:28.447233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:28.447270 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:28.448269 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:28.448271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:28.448565 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:28.448772 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:28.448777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:28.449998 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:28.449998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:28.450772 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:28.450774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:28.451421 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:28.451423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:28.451499 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:28.452068 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:28.452086 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:28.453764 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:28.456030 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:28.456037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:28.456039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:28.456361 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:28.456622 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:28.456623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:28.456624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:28.456747 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:28.457196 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:28.457197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:28.457321 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 04:49:28.457446 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 04:49:28.457637 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":58,"thread_start_us":70,"threads_started":1}
W20260810 04:49:28.458612 11291 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
W20260810 04:49:28.459206 11288 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
I20260810 04:49:28.459278 11288 leader_election.cc:302] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c93f1f1262444fb78d335310a66b7b45; no voters: 06c35cf4a5de4822badda84d6103959d, 6aed075ee04640f3b0e6aceb2beed670
I20260810 04:49:28.459388 11910 raft_consensus.cc:2726] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:49:28.522665 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:28.522481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:58613)
0810 04:49:28.522594 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:28.522633 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260810 04:49:29.314306 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:29.314061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:55983)
0810 04:49:29.314214 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:29.314259 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260810 04:49:29.398001 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:29.397807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:58787)
0810 04:49:29.397927 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:29.397963 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260810 04:49:29.523439 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:29.523227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:54133)
0810 04:49:29.523364 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:29.523403 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260810 04:49:30.315093 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:30.314866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:57011)
0810 04:49:30.315006 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:30.315046 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260810 04:49:30.398656 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:30.398438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:58807)
0810 04:49:30.398574 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:30.398619 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260810 04:49:30.399806 11929 raft_consensus.cc:482] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:30.399920 11929 raft_consensus.cc:504] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:30.400125 11929 leader_election.cc:288] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865), 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165)
W20260810 04:49:30.400558 11291 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
W20260810 04:49:30.400666 11288 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
I20260810 04:49:30.400727 11288 leader_election.cc:302] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c93f1f1262444fb78d335310a66b7b45; no voters: 06c35cf4a5de4822badda84d6103959d, 6aed075ee04640f3b0e6aceb2beed670
I20260810 04:49:30.400795 11929 raft_consensus.cc:2726] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:49:30.524104 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:30.523906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:36187)
0810 04:49:30.524031 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:30.524066 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260810 04:49:31.315891 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:31.315628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:53763)
0810 04:49:31.315779 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:31.315822 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260810 04:49:31.399322 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:31.399093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:50717)
0810 04:49:31.399239 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:31.399280 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260810 04:49:31.524760 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:31.524552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:51979)
0810 04:49:31.524678 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:31.524719 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260810 04:49:32.316757 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:32.316534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:47083)
0810 04:49:32.316665 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:32.316702 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260810 04:49:32.399982 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:32.399779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:52113)
0810 04:49:32.399905 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:32.399941 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260810 04:49:32.525470 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:32.525187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:39273)
0810 04:49:32.525391 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:32.525430 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":143,"threads_started":1}
I20260810 04:49:33.100914 11943 raft_consensus.cc:482] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:33.101065 11943 raft_consensus.cc:504] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:33.101301 11943 leader_election.cc:288] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865), 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165)
W20260810 04:49:33.101689 11291 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
W20260810 04:49:33.101814 11288 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
I20260810 04:49:33.101902 11288 leader_election.cc:302] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c93f1f1262444fb78d335310a66b7b45; no voters: 06c35cf4a5de4822badda84d6103959d, 6aed075ee04640f3b0e6aceb2beed670
I20260810 04:49:33.101982 11943 raft_consensus.cc:2726] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:49:33.317649 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:33.317397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:56791)
0810 04:49:33.317547 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:33.317587 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260810 04:49:33.400755 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:33.400546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:36667)
0810 04:49:33.400677 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:33.400712 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260810 04:49:33.526158 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:33.525887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:37211)
0810 04:49:33.526078 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:33.526118 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
I20260810 04:49:34.318395 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:34.318162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:50935)
0810 04:49:34.318304 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:34.318344 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260810 04:49:34.401412 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:34.401143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:46347)
0810 04:49:34.401332 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:34.401367 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":105,"threads_started":1}
I20260810 04:49:34.526836 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:34.526598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:37399)
0810 04:49:34.526757 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:34.526792 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260810 04:49:35.319239 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:35.318963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:39797)
0810 04:49:35.319133 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:35.319181 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260810 04:49:35.402137 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:35.401904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:37439)
0810 04:49:35.402049 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:35.402091 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260810 04:49:35.527683 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:35.527321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:59669)
0810 04:49:35.527594 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:35.527636 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":93,"threads_started":1}
I20260810 04:49:35.934742 11963 raft_consensus.cc:482] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:35.934873 11963 raft_consensus.cc:504] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:35.935056 11963 leader_election.cc:288] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865), 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165)
W20260810 04:49:35.935428 11291 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
W20260810 04:49:35.935504 11288 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
I20260810 04:49:35.935595 11288 leader_election.cc:302] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c93f1f1262444fb78d335310a66b7b45; no voters: 06c35cf4a5de4822badda84d6103959d, 6aed075ee04640f3b0e6aceb2beed670
I20260810 04:49:35.935684 11963 raft_consensus.cc:2726] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:49:36.320035 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:36.319720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:33095)
0810 04:49:36.319947 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:36.319986 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":61,"threads_started":1}
I20260810 04:49:36.402814 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:36.402590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:48541)
0810 04:49:36.402714 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:36.402750 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260810 04:49:36.528362 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:36.528138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:38523)
0810 04:49:36.528282 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:36.528319 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260810 04:49:37.320729 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:37.320512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:52575)
0810 04:49:37.320636 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:37.320667 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260810 04:49:37.403622 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:37.403291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:33505)
0810 04:49:37.403501 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:37.403559 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":126,"threads_started":1}
I20260810 04:49:37.529311 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:37.528957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:46641)
0810 04:49:37.529188 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:37.529237 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260810 04:49:38.321595 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:38.321367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:41705)
0810 04:49:38.321500 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:38.321544 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260810 04:49:38.404268 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:38.404063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:48179)
0810 04:49:38.404178 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:38.404210 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260810 04:49:38.529994 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:38.529783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:40943)
0810 04:49:38.529912 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:38.529950 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260810 04:49:39.322396 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:39.322117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:36663)
0810 04:49:39.322309 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:39.322348 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
I20260810 04:49:39.404917 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:39.404725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:33095)
0810 04:49:39.404846 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:39.404879 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260810 04:49:39.530735 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:39.530496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:59537)
0810 04:49:39.530656 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:39.530692 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260810 04:49:40.323148 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:40.322936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:40473)
0810 04:49:40.323059 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:40.323091 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260810 04:49:40.405561 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:40.405375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:47187)
0810 04:49:40.405488 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:40.405522 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260810 04:49:40.531491 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:40.531198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:39917)
0810 04:49:40.531380 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:40.531428 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260810 04:49:40.878715 11983 raft_consensus.cc:482] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:40.878849 11983 raft_consensus.cc:504] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed075ee04640f3b0e6aceb2beed670" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43865 } } peers { permanent_uuid: "06c35cf4a5de4822badda84d6103959d" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 40165 } } peers { permanent_uuid: "c93f1f1262444fb78d335310a66b7b45" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 43207 } }
I20260810 04:49:40.879098 11983 leader_election.cc:288] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865), 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165)
W20260810 04:49:40.879501 11288 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6aed075ee04640f3b0e6aceb2beed670 (127.14.237.3:43865): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
W20260810 04:49:40.879587 11291 leader_election.cc:341] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06c35cf4a5de4822badda84d6103959d (127.14.237.2:40165): Not found: Tablet not found: 4a6ae54a56c944a6b7ce01c6f3b9ab10
I20260810 04:49:40.879666 11291 leader_election.cc:302] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c93f1f1262444fb78d335310a66b7b45; no voters: 06c35cf4a5de4822badda84d6103959d, 6aed075ee04640f3b0e6aceb2beed670
I20260810 04:49:40.879765 11983 raft_consensus.cc:2726] T 4a6ae54a56c944a6b7ce01c6f3b9ab10 P c93f1f1262444fb78d335310a66b7b45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:49:41.323881 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:41.323645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:38461)
0810 04:49:41.323790 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:41.323831 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260810 04:49:41.406257 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:41.406034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:44953)
0810 04:49:41.406185 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:41.406219 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260810 04:49:41.532152 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:41.531916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:41201)
0810 04:49:41.532049 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:41.532084 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260810 04:49:42.324736 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:42.324502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.1:56293)
0810 04:49:42.324642 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:42.324677 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260810 04:49:42.406899 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:42.406718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.2:47815)
0810 04:49:42.406831 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:42.406862 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260810 04:49:42.526887 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 11212
I20260810 04:49:42.534018 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:42.533397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38863 (local address 127.14.237.3:35933)
0810 04:49:42.533927 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:42.533978 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.62:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":59,"threads_started":1}
I20260810 04:49:42.538255 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 11496
I20260810 04:49:42.551765 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 11700
2026-08-10T04:49:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58658 ms)
[----------] 30 tests from SecurityITest (237201 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:42 dist-test-slave-8hhm krb5kdc[11997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:42 dist-test-slave-8hhm krb5kdc[11997](info): set up 2 sockets
Aug 10 04:49:42 dist-test-slave-8hhm krb5kdc[11997](info): commencing operation
krb5kdc: starting...
W20260810 04:49:43.816290 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:49:43 dist-test-slave-8hhm krb5kdc[11997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:49:43.940331 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:41613
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:44.020792 12016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:44.020978 12016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:44.020998 12016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:44.022025 12016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:44.022063 12016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:44.022078 12016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:44.022109 12016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:44.022120 12016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:44.023538 12016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:41613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:44.023723 12016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:44.023880 12016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:44.029032 12025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:44.029055 12023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:44.029055 12032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:44.029074 12016 server_base.cc:1034] running on GCE node
I20260810 04:49:44.029660 12016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:44.029953 12016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:44.031131 12016 hybrid_clock.cc:648] HybridClock initialized: now 1786337384031114 us; error 26 us; skew 500 ppm
Aug 10 04:49:44 dist-test-slave-8hhm krb5kdc[11997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337384, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:44.033746 12016 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:49:44.034211 12016 webserver.cc:460] Webserver started at http://127.14.237.62:33201/ using document root <none> and password file <none>
I20260810 04:49:44.034449 12016 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:44.034524 12016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:44.034633 12016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:44.035670 12016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "5cb681f052a04a21957b0cc39f9e8bce"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "16f075d4a551aa71647cea70c801053b"
server_key_iv: "a39ea39cba5e33d02ff976ad2ccff02c"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.035981 12016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "5cb681f052a04a21957b0cc39f9e8bce"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "16f075d4a551aa71647cea70c801053b"
server_key_iv: "a39ea39cba5e33d02ff976ad2ccff02c"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.037323 12016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 04:49:44.038189 12045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.038347 12016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:44.038436 12016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "5cb681f052a04a21957b0cc39f9e8bce"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "16f075d4a551aa71647cea70c801053b"
server_key_iv: "a39ea39cba5e33d02ff976ad2ccff02c"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.038491 12016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:44.047497 12016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:44.049156 12016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:44.049329 12016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:44.053090 12016 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:41613
I20260810 04:49:44.053153 12125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:41613 every 8 connection(s)
I20260810 04:49:44.053476 12016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:44.054356 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:44.054936 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 12016
I20260810 04:49:44.055068 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:44.055269 15284 external_mini_cluster.cc:1428] Setting key 3cda5ffe8f7b805b4e56c05ae22b2f11
I20260810 04:49:44.058055 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce: Bootstrap starting.
I20260810 04:49:44.058794 12126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:44.059091 12126 log.cc:824] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:44.059852 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce: No bootstrap required, opened a new log
Aug 10 04:49:44 dist-test-slave-8hhm krb5kdc[11997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:44.062021 12126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb681f052a04a21957b0cc39f9e8bce" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41613 } }
I20260810 04:49:44.062238 12126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:44.062294 12126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb681f052a04a21957b0cc39f9e8bce, State: Initialized, Role: FOLLOWER
I20260810 04:49:44.062422 12126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb681f052a04a21957b0cc39f9e8bce" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41613 } }
I20260810 04:49:44.062500 12126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:44.062546 12126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:44.062592 12126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:44.063273 12126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb681f052a04a21957b0cc39f9e8bce" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41613 } }
I20260810 04:49:44.063398 12126 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb681f052a04a21957b0cc39f9e8bce; no voters:
I20260810 04:49:44.063570 12126 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:44.063647 12136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:44.063822 12126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:44.063858 12136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [term 1 LEADER]: Becoming Leader. State: Replica: 5cb681f052a04a21957b0cc39f9e8bce, State: Running, Role: LEADER
I20260810 04:49:44.064101 12136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb681f052a04a21957b0cc39f9e8bce" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41613 } }
I20260810 04:49:44.064430 12136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb681f052a04a21957b0cc39f9e8bce. Latest consensus state: current_term: 1 leader_uuid: "5cb681f052a04a21957b0cc39f9e8bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb681f052a04a21957b0cc39f9e8bce" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41613 } } }
I20260810 04:49:44.064563 12138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb681f052a04a21957b0cc39f9e8bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb681f052a04a21957b0cc39f9e8bce" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 41613 } } }
I20260810 04:49:44.064630 12136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:44.064831 12138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:44.065375 12145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:44.065963 12145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:44.066749 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.055807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.14.237.62:41613)
0810 04:49:44.055986 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.055993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.056018 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:49:44.056423 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.056429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.056473 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.056582 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:44.057427 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.058322 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.059433 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.059899 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.060024 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.062533 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:44.062537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:44.062538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:44.062550 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:44.064618 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.065054 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.065056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.065057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.065163 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.065467 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.065468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.065469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.065762 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:44.066333 (+ 571us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.066402 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.066534 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260810 04:49:44.067924 12145 catalog_manager.cc:1347] Generated new cluster ID: 32309536badf4949ac820aa324e60165
I20260810 04:49:44.067982 12145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:44.083261 12145 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:44.083791 12145 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:44.087507 12145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce: Generated new TSK 0
I20260810 04:49:44.087688 12145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:49:44.120307 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:41613
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:44.202730 12170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:44.202922 12170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:44.202946 12170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:44.203977 12170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:44.204010 12170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:44.204035 12170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:49:44.205528 12170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:44.205703 12170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:44.205873 12170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:44.210711 12178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:44.210749 12181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:44.210743 12179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:44.211023 12170 server_base.cc:1034] running on GCE node
I20260810 04:49:44.211221 12170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:44.211442 12170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:44.212610 12170 hybrid_clock.cc:648] HybridClock initialized: now 1786337384212605 us; error 34 us; skew 500 ppm
Aug 10 04:49:44 dist-test-slave-8hhm krb5kdc[11997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337384, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:44.215099 12170 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:49:44.215548 12170 webserver.cc:460] Webserver started at http://127.14.237.1:32789/ using document root <none> and password file <none>
I20260810 04:49:44.215773 12170 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:44.215842 12170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:44.215979 12170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:44.216967 12170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "4abbd00822bc423588399c70ff31d696"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "3d67c83a79af101e1e1163ebc3f8b3f7"
server_key_iv: "ee1560cd5a5d5e22b1ec467618c29dc5"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.217355 12170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "4abbd00822bc423588399c70ff31d696"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "3d67c83a79af101e1e1163ebc3f8b3f7"
server_key_iv: "ee1560cd5a5d5e22b1ec467618c29dc5"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.218694 12170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:44.219455 12191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.219646 12170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:44.219745 12170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "4abbd00822bc423588399c70ff31d696"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "3d67c83a79af101e1e1163ebc3f8b3f7"
server_key_iv: "ee1560cd5a5d5e22b1ec467618c29dc5"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.219831 12170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:44.236380 12170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:44.238714 12170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:44.238886 12170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:44.239130 12170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:44.239449 12170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:44.239503 12170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.239545 12170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:44.239573 12170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.245769 12170 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:38231
I20260810 04:49:44.245813 12365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:38231 every 8 connection(s)
I20260810 04:49:44.246184 12170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:49:44 dist-test-slave-8hhm krb5kdc[11997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337384, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:44.255676 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 12170
I20260810 04:49:44.255805 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:49:44.256007 15284 external_mini_cluster.cc:1428] Setting key 174de21053853a34343b49c1e9d299dd
I20260810 04:49:44.256542 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.247107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:43323 (local address 127.14.237.62:41613)
0810 04:49:44.247225 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.247228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.247429 (+ 201us) server_negotiation.cc:407] Connection header received
0810 04:49:44.248014 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.248015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.248029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.248085 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:44.249167 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.249759 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.251032 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.251256 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.251352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.253680 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:44.253682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:44.253682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:44.253690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:44.255108 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.255485 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.255486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.255486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.255540 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.255856 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.255857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.255858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.255985 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:44.256067 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.256362 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.256452 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 04:49:44.256738 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.246950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:41613 (local address 127.14.237.1:43323)
0810 04:49:44.247266 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.247331 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:44.247893 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:44.248191 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:44.248195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:44.248811 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:44.249022 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.249028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.249880 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.249881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.250906 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.250910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.251373 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.251375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.251506 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.252089 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:44.252099 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:44.253585 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:44.255187 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:44.255193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:44.255195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:44.255416 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:44.255640 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:44.255641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:44.255641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:44.255757 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:44.256067 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:44.256075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:44.256218 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 04:49:44.256324 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:49:44.256487 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":144,"threads_started":1}
I20260810 04:49:44.257519 12367 heartbeater.cc:347] Connected to a master server at 127.14.237.62:41613
I20260810 04:49:44.257625 12367 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:44.257822 12367 heartbeater.cc:511] Master 127.14.237.62:41613 requested a full tablet report, sending...
I20260810 04:49:44.258383 12067 ts_manager.cc:194] Registered new tserver with Master: 4abbd00822bc423588399c70ff31d696 (127.14.237.1:38231)
I20260810 04:49:44.258982 12067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:43323
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:49:44.283659 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:41613
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:44.365619 12373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:44.365856 12373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:44.365895 12373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:44.367002 12373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:44.367076 12373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:44.367126 12373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:49:44.368630 12373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:44.368906 12373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:44.369151 12373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:44.374329 12381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:44.374410 12385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:44.374337 12382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:44.374706 12373 server_base.cc:1034] running on GCE node
I20260810 04:49:44.374926 12373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:44.375177 12373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:44.376344 12373 hybrid_clock.cc:648] HybridClock initialized: now 1786337384376343 us; error 40 us; skew 500 ppm
Aug 10 04:49:44 dist-test-slave-8hhm krb5kdc[11997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337384, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:44.378904 12373 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:49:44.379347 12373 webserver.cc:460] Webserver started at http://127.14.237.2:46193/ using document root <none> and password file <none>
I20260810 04:49:44.379597 12373 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:44.379666 12373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:44.379817 12373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:44.380837 12373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "952d8611430d4e8f81926cacddeb0056"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "e660f4474884ff4537c6693ade9eb983"
server_key_iv: "fa98c93b6fc6d8640e6037146712154c"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.381203 12373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "952d8611430d4e8f81926cacddeb0056"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "e660f4474884ff4537c6693ade9eb983"
server_key_iv: "fa98c93b6fc6d8640e6037146712154c"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.382620 12373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:44.383399 12398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.383579 12373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:44.383682 12373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "952d8611430d4e8f81926cacddeb0056"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "e660f4474884ff4537c6693ade9eb983"
server_key_iv: "fa98c93b6fc6d8640e6037146712154c"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.383769 12373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:44.395344 12373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:44.397682 12373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:44.397850 12373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:44.398074 12373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:44.398384 12373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:44.398437 12373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.398469 12373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:44.398495 12373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.404271 12373 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:44789
I20260810 04:49:44.404312 12575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:44789 every 8 connection(s)
I20260810 04:49:44.404683 12373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:49:44.408847 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 12373
I20260810 04:49:44.408954 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:49:44.409173 15284 external_mini_cluster.cc:1428] Setting key cc4ade6d62aed56f1dec4310f4b493a9
Aug 10 04:49:44 dist-test-slave-8hhm krb5kdc[11997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337384, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:44.414666 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.405461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:46651 (local address 127.14.237.62:41613)
0810 04:49:44.405520 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.405522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.405720 (+ 198us) server_negotiation.cc:407] Connection header received
0810 04:49:44.406193 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.406194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.406204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.406255 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:44.407153 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.407733 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.408699 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.408998 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.409074 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.411411 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:44.411412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:44.411412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:44.411421 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:44.412998 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.413616 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.413617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.413618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.413684 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.414012 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.414013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.414014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.414160 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:44.414234 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.414498 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.414574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:49:44.414811 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.405413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:41613 (local address 127.14.237.2:46651)
0810 04:49:44.405578 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.405618 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:44.406106 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:44.406314 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:44.406317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:44.406858 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:44.407067 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.407071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.407815 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.407816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.408598 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.408600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.409089 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.409090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.409192 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.409808 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:44.409818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:44.411318 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:44.413097 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:44.413110 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:44.413112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:44.413495 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:44.413760 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:44.413760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:44.413761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:44.413936 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:44.414238 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:44.414244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:44.414377 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:49:44.414474 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 04:49:44.414606 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260810 04:49:44.415396 12576 heartbeater.cc:347] Connected to a master server at 127.14.237.62:41613
I20260810 04:49:44.415472 12576 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:44.415650 12576 heartbeater.cc:511] Master 127.14.237.62:41613 requested a full tablet report, sending...
I20260810 04:49:44.416086 12067 ts_manager.cc:194] Registered new tserver with Master: 952d8611430d4e8f81926cacddeb0056 (127.14.237.2:44789)
I20260810 04:49:44.416649 12067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:46651
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:49:44.435961 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:41613
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:44.518462 12582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:44.518677 12582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:44.518702 12582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:44.519738 12582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:44.519794 12582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:44.519821 12582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:49:44.521252 12582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:44.521435 12582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:44.521625 12582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:44.526619 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:44.526604 12587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:44.526598 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:44.526829 12582 server_base.cc:1034] running on GCE node
I20260810 04:49:44.527030 12582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:44.527261 12582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:44.528445 12582 hybrid_clock.cc:648] HybridClock initialized: now 1786337384528432 us; error 24 us; skew 500 ppm
Aug 10 04:49:44 dist-test-slave-8hhm krb5kdc[11997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337384, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:44.531111 12582 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:49:44.531548 12582 webserver.cc:460] Webserver started at http://127.14.237.3:46597/ using document root <none> and password file <none>
I20260810 04:49:44.531785 12582 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:44.531840 12582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:44.531984 12582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:44.533061 12582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "0b4d45bc57144c4a8312c4668fc90c26"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "22e06e9b7e1b8596f07235ca31b53a94"
server_key_iv: "f374ae19c74c046c0be1f474bad06309"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.533465 12582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "0b4d45bc57144c4a8312c4668fc90c26"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "22e06e9b7e1b8596f07235ca31b53a94"
server_key_iv: "f374ae19c74c046c0be1f474bad06309"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.534740 12582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:44.535458 12602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.535633 12582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:44.535733 12582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "0b4d45bc57144c4a8312c4668fc90c26"
format_stamp: "Formatted at 2026-08-10 04:49:44 on dist-test-slave-8hhm"
server_key: "22e06e9b7e1b8596f07235ca31b53a94"
server_key_iv: "f374ae19c74c046c0be1f474bad06309"
server_key_version: "encryptionkey@0"
I20260810 04:49:44.535801 12582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:44.547096 12582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:44.549386 12582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:44.549541 12582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:44.549760 12582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:44.550110 12582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:44.550160 12582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.550189 12582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:44.550210 12582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:44.556342 12582 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:33219
I20260810 04:49:44.556403 12793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:33219 every 8 connection(s)
I20260810 04:49:44.556704 12582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:49:44.561326 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 12582
I20260810 04:49:44.561455 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:49:44.561676 15284 external_mini_cluster.cc:1428] Setting key 08ca44b15431afbcda581fe01b9f10be
Aug 10 04:49:44 dist-test-slave-8hhm krb5kdc[11997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337384, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:44.566612 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.557479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:37361 (local address 127.14.237.62:41613)
0810 04:49:44.557580 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.557582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.557730 (+ 148us) server_negotiation.cc:407] Connection header received
0810 04:49:44.558254 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.558255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.558267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.558315 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:44.559293 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.559900 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.560850 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.561067 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.561145 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.563544 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:44.563545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:44.563545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:44.563552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:44.565045 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.565588 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.565590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.565590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.565652 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.565911 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.565912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.565912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.566028 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:44.566102 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.566475 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.566548 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:49:44.566835 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.557446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:41613 (local address 127.14.237.3:37361)
0810 04:49:44.557607 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.557647 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:44.558159 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:44.558389 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:44.558393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:44.558908 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:44.559176 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.559181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.559992 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.559993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.560757 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.560759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.561153 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.561154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.561327 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.561842 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:44.561850 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:44.563442 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:44.565160 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:44.565167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:44.565169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:44.565470 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:44.565722 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:44.565723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:44.565723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:44.565835 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:44.566104 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:44.566111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:44.566267 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 04:49:44.566417 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 04:49:44.566590 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260810 04:49:44.567332 12795 heartbeater.cc:347] Connected to a master server at 127.14.237.62:41613
I20260810 04:49:44.567411 12795 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:44.567541 12795 heartbeater.cc:511] Master 127.14.237.62:41613 requested a full tablet report, sending...
I20260810 04:49:44.567924 12067 ts_manager.cc:194] Registered new tserver with Master: 0b4d45bc57144c4a8312c4668fc90c26 (127.14.237.3:33219)
I20260810 04:49:44.568239 12067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:37361
I20260810 04:49:44.573458 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:49:44.580447 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.574782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.14.237.62:41613)
0810 04:49:44.574847 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.574849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.574944 (+ 95us) server_negotiation.cc:407] Connection header received
0810 04:49:44.575017 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.575018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.575035 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.575137 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:44.575677 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.576124 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.576888 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.577093 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.577166 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.578253 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:44.578255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:44.578255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:44.578262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:44.579330 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.579691 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.579692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.579692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.579744 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:44.579927 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:44.579928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:44.579928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:44.580083 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:44.580196 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.580286 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.580357 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:49:44.581804 12067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:49:44.583364 12067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:49:44.589541 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.585638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:38231 (local address 127.0.0.1:44210)
0810 04:49:44.585744 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.585755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:44.585813 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:44.586188 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:44.586189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:44.586191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:44.586248 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.586250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.587218 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.587219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.588507 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.588510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.589207 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.589208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.589308 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.589327 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 04:49:44.589414 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:49:44.589471 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":16,"thread_start_us":54,"threads_started":1}
I20260810 04:49:44.589550 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.585814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.14.237.1:38231)
0810 04:49:44.586015 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.586020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.586029 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:49:44.586070 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.586071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.586087 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.586325 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:44.586403 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.587127 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.588603 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.589102 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.589195 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.589284 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.589432 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.589477 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260810 04:49:44.589701 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.585425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:33219 (local address 127.0.0.1:59766)
0810 04:49:44.585751 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.585760 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:44.585840 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:44.586123 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:44.586126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:44.586128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:44.586190 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.586192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.587542 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.587543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.588567 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.588569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.589470 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.589471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.589535 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.589538 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:44.589567 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 04:49:44.589609 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":55,"threads_started":1}
I20260810 04:49:44.589931 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.585433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:44789 (local address 127.0.0.1:54456)
0810 04:49:44.585684 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.585724 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:44.585784 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:44.586241 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:44.586242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:44.586243 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:44.586279 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.586281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.587569 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.587570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.588537 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.588540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.589521 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.589521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.589576 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.589579 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:44.589833 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 04:49:44.589876 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":65,"threads_started":1}
I20260810 04:49:44.590070 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.585509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.14.237.3:33219)
0810 04:49:44.585652 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.585658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.585816 (+ 158us) server_negotiation.cc:407] Connection header received
0810 04:49:44.585931 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.585934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.585948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.586269 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:44.586372 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.587452 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.588652 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.589368 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.589875 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.589936 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.589962 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.590002 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260810 04:49:44.590334 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.585547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.14.237.2:44789)
0810 04:49:44.586015 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.586019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.586029 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:49:44.586069 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.586071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.586087 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.586163 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:44.586345 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.587452 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.588641 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.589368 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.589439 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.589493 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.590030 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.590071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":82,"threads_started":1}
I20260810 04:49:44.590361 12265 tablet_service.cc:1467] Processing CreateTablet for tablet ae8a0d1aafc049acbc9ee308d0c65880 (DEFAULT_TABLE table=test-table [id=db1a8f43e8bd4733adf303949ec9c41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:44.590655 12689 tablet_service.cc:1467] Processing CreateTablet for tablet ae8a0d1aafc049acbc9ee308d0c65880 (DEFAULT_TABLE table=test-table [id=db1a8f43e8bd4733adf303949ec9c41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:44.590771 12265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8a0d1aafc049acbc9ee308d0c65880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:44.590986 12689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8a0d1aafc049acbc9ee308d0c65880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:44.593341 12815 tablet_bootstrap.cc:492] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696: Bootstrap starting.
I20260810 04:49:44.593470 12816 tablet_bootstrap.cc:492] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Bootstrap starting.
I20260810 04:49:44.594056 12816 tablet_bootstrap.cc:654] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:44.594122 12815 tablet_bootstrap.cc:654] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:44.594022 12479 tablet_service.cc:1467] Processing CreateTablet for tablet ae8a0d1aafc049acbc9ee308d0c65880 (DEFAULT_TABLE table=test-table [id=db1a8f43e8bd4733adf303949ec9c41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:44.594380 12479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8a0d1aafc049acbc9ee308d0c65880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:44.594655 12816 log.cc:824] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:44.594679 12815 log.cc:824] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:44.595431 12815 tablet_bootstrap.cc:492] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696: No bootstrap required, opened a new log
I20260810 04:49:44.595443 12816 tablet_bootstrap.cc:492] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: No bootstrap required, opened a new log
I20260810 04:49:44.595505 12815 ts_tablet_manager.cc:1397] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:49:44.595525 12816 ts_tablet_manager.cc:1397] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:44.596906 12816 raft_consensus.cc:348] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.597066 12816 raft_consensus.cc:374] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:44.597110 12816 raft_consensus.cc:729] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4d45bc57144c4a8312c4668fc90c26, State: Initialized, Role: FOLLOWER
I20260810 04:49:44.597275 12816 consensus_queue.cc:260] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.597532 12815 raft_consensus.cc:348] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.597707 12815 raft_consensus.cc:374] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:44.597750 12815 raft_consensus.cc:729] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4abbd00822bc423588399c70ff31d696, State: Initialized, Role: FOLLOWER
I20260810 04:49:44.597759 12816 ts_tablet_manager.cc:1428] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:44.597880 12795 heartbeater.cc:503] Master 127.14.237.62:41613 was elected leader, sending a full tablet report...
I20260810 04:49:44.597874 12820 tablet_bootstrap.cc:492] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Bootstrap starting.
I20260810 04:49:44.597849 12815 consensus_queue.cc:260] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.598148 12815 ts_tablet_manager.cc:1428] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 04:49:44.598165 12367 heartbeater.cc:503] Master 127.14.237.62:41613 was elected leader, sending a full tablet report...
I20260810 04:49:44.598572 12820 tablet_bootstrap.cc:654] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:44.599148 12820 log.cc:824] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:44.599824 12820 tablet_bootstrap.cc:492] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: No bootstrap required, opened a new log
I20260810 04:49:44.599874 12820 ts_tablet_manager.cc:1397] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:44.601353 12820 raft_consensus.cc:348] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.601482 12820 raft_consensus.cc:374] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:44.601503 12820 raft_consensus.cc:729] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952d8611430d4e8f81926cacddeb0056, State: Initialized, Role: FOLLOWER
I20260810 04:49:44.601605 12820 consensus_queue.cc:260] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.601812 12820 ts_tablet_manager.cc:1428] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:49:44.601881 12576 heartbeater.cc:503] Master 127.14.237.62:41613 was elected leader, sending a full tablet report...
W20260810 04:49:44.655380 12578 tablet.cc:2369] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:49:44.747107 12369 tablet.cc:2369] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:49:44.807351 12796 tablet.cc:2369] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:49:44.834795 12825 raft_consensus.cc:482] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:44.834925 12825 raft_consensus.cc:504] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.835354 12825 leader_election.cc:288] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b4d45bc57144c4a8312c4668fc90c26 (127.14.237.3:33219), 4abbd00822bc423588399c70ff31d696 (127.14.237.1:38231)
I20260810 04:49:44.838260 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.835703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:40427 (local address 127.14.237.1:38231)
0810 04:49:44.835753 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.835754 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.835763 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:49:44.835803 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.835804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.835815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.835892 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:44.836050 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.836526 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.837508 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.837896 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.837984 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.837994 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.838152 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.838188 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:49:44.838260 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.835589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:38231 (local address 127.14.237.2:40427)
0810 04:49:44.835649 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.835662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:44.835731 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:44.835901 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:44.835902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:44.835905 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:44.835978 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.835979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.836622 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.836623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.837407 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.837410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.837983 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.837984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.838071 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.838075 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:44.838144 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:49:44.838187 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 04:49:44.838507 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.835589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:33219 (local address 127.14.237.2:35197)
0810 04:49:44.835861 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.835878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:44.835941 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:44.836087 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:44.836088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:44.836089 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:44.836166 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.836167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.836771 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.836773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.837587 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:44.837589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.838200 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.838201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:44.838282 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.838286 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:44.838359 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:49:44.838404 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":91,"thread_start_us":130,"threads_started":1}
I20260810 04:49:44.838609 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.835700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:35197 (local address 127.14.237.3:33219)
0810 04:49:44.835749 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.835751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.835945 (+ 194us) server_negotiation.cc:407] Connection header received
0810 04:49:44.835995 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.835996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.836005 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.836068 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:44.836245 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.836685 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.837677 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.838114 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.838200 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.838208 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.838487 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.838530 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:49:44.838598 12301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8a0d1aafc049acbc9ee308d0c65880" candidate_uuid: "952d8611430d4e8f81926cacddeb0056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4abbd00822bc423588399c70ff31d696" is_pre_election: true
I20260810 04:49:44.838786 12301 raft_consensus.cc:2454] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952d8611430d4e8f81926cacddeb0056 in term 0.
I20260810 04:49:44.838990 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8a0d1aafc049acbc9ee308d0c65880" candidate_uuid: "952d8611430d4e8f81926cacddeb0056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4d45bc57144c4a8312c4668fc90c26" is_pre_election: true
I20260810 04:49:44.839058 12400 leader_election.cc:302] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abbd00822bc423588399c70ff31d696, 952d8611430d4e8f81926cacddeb0056; no voters:
I20260810 04:49:44.839150 12722 raft_consensus.cc:2454] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952d8611430d4e8f81926cacddeb0056 in term 0.
I20260810 04:49:44.839184 12825 raft_consensus.cc:2781] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:49:44.839236 12825 raft_consensus.cc:482] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:49:44.839253 12825 raft_consensus.cc:3037] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:44.840013 12825 raft_consensus.cc:504] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.840169 12825 leader_election.cc:288] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [CANDIDATE]: Term 1 election: Requested vote from peers 0b4d45bc57144c4a8312c4668fc90c26 (127.14.237.3:33219), 4abbd00822bc423588399c70ff31d696 (127.14.237.1:38231)
I20260810 04:49:44.840344 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8a0d1aafc049acbc9ee308d0c65880" candidate_uuid: "952d8611430d4e8f81926cacddeb0056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4d45bc57144c4a8312c4668fc90c26"
I20260810 04:49:44.840376 12301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8a0d1aafc049acbc9ee308d0c65880" candidate_uuid: "952d8611430d4e8f81926cacddeb0056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4abbd00822bc423588399c70ff31d696"
I20260810 04:49:44.840428 12722 raft_consensus.cc:3037] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:44.840440 12301 raft_consensus.cc:3037] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:44.841616 12722 raft_consensus.cc:2454] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952d8611430d4e8f81926cacddeb0056 in term 1.
I20260810 04:49:44.841616 12301 raft_consensus.cc:2454] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952d8611430d4e8f81926cacddeb0056 in term 1.
I20260810 04:49:44.841820 12400 leader_election.cc:302] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abbd00822bc423588399c70ff31d696, 952d8611430d4e8f81926cacddeb0056; no voters:
I20260810 04:49:44.841939 12825 raft_consensus.cc:2781] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:44.842111 12825 raft_consensus.cc:686] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 1 LEADER]: Becoming Leader. State: Replica: 952d8611430d4e8f81926cacddeb0056, State: Running, Role: LEADER
I20260810 04:49:44.842231 12825 consensus_queue.cc:237] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } }
I20260810 04:49:44.843132 12066 catalog_manager.cc:5495] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 reported cstate change: term changed from 0 to 1, leader changed from <none> to 952d8611430d4e8f81926cacddeb0056 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "952d8611430d4e8f81926cacddeb0056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "952d8611430d4e8f81926cacddeb0056" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 44789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 } health_report { overall_health: UNKNOWN } } }
I20260810 04:49:44.875427 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.873038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.14.237.2:44789)
0810 04:49:44.873097 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:44.873099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:44.873123 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:49:44.873159 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:44.873160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:44.873177 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:44.873305 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:49:44.873438 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.873891 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.874693 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:44.874881 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:44.874933 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:44.875097 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:49:44.875218 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:49:44.875284 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:44.875323 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 04:49:44.875340 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:49:44.877354 12722 raft_consensus.cc:1261] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 1 FOLLOWER]: Refusing update from remote peer 952d8611430d4e8f81926cacddeb0056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:44.877349 12301 raft_consensus.cc:1261] T ae8a0d1aafc049acbc9ee308d0c65880 P 4abbd00822bc423588399c70ff31d696 [term 1 FOLLOWER]: Refusing update from remote peer 952d8611430d4e8f81926cacddeb0056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:44.877765 12825 consensus_queue.cc:1035] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4abbd00822bc423588399c70ff31d696" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:44.877923 12828 consensus_queue.cc:1035] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4d45bc57144c4a8312c4668fc90c26" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 33219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:44.882674 12066 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:44.882789 12066 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:44.890255 12066 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cb681f052a04a21957b0cc39f9e8bce: Sending DeleteTablet for 3 replicas of tablet ae8a0d1aafc049acbc9ee308d0c65880
I20260810 04:49:44.890676 12265 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8a0d1aafc049acbc9ee308d0c65880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:44 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:44210
I20260810 04:49:44.891129 12689 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8a0d1aafc049acbc9ee308d0c65880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:44 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:59766
I20260810 04:49:44.891677 12479 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8a0d1aafc049acbc9ee308d0c65880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:44 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:54456
I20260810 04:49:44.891806 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 12170
I20260810 04:49:44.898722 12852 tablet_replica.cc:330] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: stopping tablet replica
I20260810 04:49:44.898840 12852 raft_consensus.cc:2229] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:44.898939 12852 raft_consensus.cc:2258] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:44.899456 12852 ts_tablet_manager.cc:1905] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:44.901340 12852 ts_tablet_manager.cc:1918] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:44.901405 12852 log.cc:1196] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/ae8a0d1aafc049acbc9ee308d0c65880
I20260810 04:49:44.901679 12852 ts_tablet_manager.cc:1939] T ae8a0d1aafc049acbc9ee308d0c65880 P 0b4d45bc57144c4a8312c4668fc90c26: Deleting consensus metadata
I20260810 04:49:44.902067 12050 catalog_manager.cc:4841] TS 0b4d45bc57144c4a8312c4668fc90c26 (127.14.237.3:33219): tablet ae8a0d1aafc049acbc9ee308d0c65880 (table test-table [id=db1a8f43e8bd4733adf303949ec9c41c]) successfully deleted
I20260810 04:49:44.909368 12854 tablet_replica.cc:330] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: stopping tablet replica
I20260810 04:49:44.909489 12854 raft_consensus.cc:2229] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:49:44.909597 12854 raft_consensus.cc:2258] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:44.910372 12854 ts_tablet_manager.cc:1905] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:44.912117 12854 ts_tablet_manager.cc:1918] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:44.912185 12854 log.cc:1196] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/ae8a0d1aafc049acbc9ee308d0c65880
I20260810 04:49:44.912441 12854 ts_tablet_manager.cc:1939] T ae8a0d1aafc049acbc9ee308d0c65880 P 952d8611430d4e8f81926cacddeb0056: Deleting consensus metadata
W20260810 04:49:44.912528 12047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:38231 (error 108)
I20260810 04:49:44.912835 12051 catalog_manager.cc:4841] TS 952d8611430d4e8f81926cacddeb0056 (127.14.237.2:44789): tablet ae8a0d1aafc049acbc9ee308d0c65880 (table test-table [id=db1a8f43e8bd4733adf303949ec9c41c]) successfully deleted
I20260810 04:49:44.913105 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:44.912967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:38231 (local address 127.0.0.1:44214)
0810 04:49:44.913015 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:44.913066 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:38231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260810 04:49:44.913254 12047 catalog_manager.cc:4584] TS 4abbd00822bc423588399c70ff31d696 (127.14.237.1:38231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae8a0d1aafc049acbc9ee308d0c65880: Network error: Client connection negotiation failed: client connection to 127.14.237.1:38231: connect: Connection refused (error 111)
I20260810 04:49:44.913368 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 12373
I20260810 04:49:44.924830 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 12582
I20260810 04:49:44.937963 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 12016
2026-08-10T04:49:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2380 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T04:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:44Z Disabled control of system clock
I20260810 04:49:44.971028 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:44573
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:44573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:45.056703 12864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:45.056957 12864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:45.056993 12864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:45.058216 12864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:45.058283 12864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:45.058303 12864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:45.058324 12864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:45.058351 12864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:45.059746 12864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:44573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:44573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:45.060030 12864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:45.060421 12864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:45.067194 12872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:45.067283 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:45.067382 12864 server_base.cc:1034] running on GCE node
W20260810 04:49:45.067534 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:45.067852 12864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:45.070307 12864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:45.071527 12864 hybrid_clock.cc:648] HybridClock initialized: now 1786337385071520 us; error 48 us; skew 500 ppm
I20260810 04:49:45.072985 12864 webserver.cc:460] Webserver started at http://127.14.237.62:43303/ using document root <none> and password file <none>
I20260810 04:49:45.073318 12864 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:45.073387 12864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:45.073540 12864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:45.074609 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "5c4e4d7d55c342caba76191771e9f7c6"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "1082cec836f3eed76a9765f29476b081"
server_key_iv: "14e8cbab593e2af9f1e506ef809e1ad0"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.074993 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "5c4e4d7d55c342caba76191771e9f7c6"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "1082cec836f3eed76a9765f29476b081"
server_key_iv: "14e8cbab593e2af9f1e506ef809e1ad0"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.076520 12864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:49:45.077450 12883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.077632 12864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:45.077747 12864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "5c4e4d7d55c342caba76191771e9f7c6"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "1082cec836f3eed76a9765f29476b081"
server_key_iv: "14e8cbab593e2af9f1e506ef809e1ad0"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.077837 12864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:45.087678 12864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:45.087982 12864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:45.088130 12864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:45.092022 12864 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:44573
I20260810 04:49:45.092068 12969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:44573 every 8 connection(s)
I20260810 04:49:45.092429 12864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:45.093501 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:45.095615 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 12864
I20260810 04:49:45.095741 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:45.095887 12971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6: Bootstrap starting.
I20260810 04:49:45.095955 15284 external_mini_cluster.cc:1428] Setting key 3aa8e4e21cd9c4fd40bd4fd8be5c9aab
I20260810 04:49:45.096587 12971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:45.096902 12971 log.cc:824] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:45.097649 12971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6: No bootstrap required, opened a new log
I20260810 04:49:45.099396 12971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e4d7d55c342caba76191771e9f7c6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44573 } }
I20260810 04:49:45.099525 12971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:45.099547 12971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4e4d7d55c342caba76191771e9f7c6, State: Initialized, Role: FOLLOWER
I20260810 04:49:45.099689 12971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e4d7d55c342caba76191771e9f7c6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44573 } }
I20260810 04:49:45.099771 12971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:45.099790 12971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:45.099854 12971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:45.100387 12971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e4d7d55c342caba76191771e9f7c6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44573 } }
I20260810 04:49:45.100519 12971 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c4e4d7d55c342caba76191771e9f7c6; no voters:
I20260810 04:49:45.100698 12971 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:45.100744 12977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:45.100867 12977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5c4e4d7d55c342caba76191771e9f7c6, State: Running, Role: LEADER
I20260810 04:49:45.100980 12971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:45.100960 12977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e4d7d55c342caba76191771e9f7c6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44573 } }
I20260810 04:49:45.101331 12977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4e4d7d55c342caba76191771e9f7c6. Latest consensus state: current_term: 1 leader_uuid: "5c4e4d7d55c342caba76191771e9f7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e4d7d55c342caba76191771e9f7c6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44573 } } }
I20260810 04:49:45.101416 12977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:45.101346 12979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c4e4d7d55c342caba76191771e9f7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4e4d7d55c342caba76191771e9f7c6" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44573 } } }
I20260810 04:49:45.101575 12979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:45.101648 12982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:45.101763 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.096460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.14.237.62:44573)
0810 04:49:45.096688 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.096697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.096732 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:49:45.097205 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.097249 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.097325 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.097461 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:45.097681 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.098559 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.099328 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.099881 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.099992 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.100738 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:45.100742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:45.100742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:45.100955 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:45.100996 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:45.101009 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:45.101340 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.101375 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.101499 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260810 04:49:45.102159 12982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:45.106535 12982 catalog_manager.cc:1347] Generated new cluster ID: 62c5f843ba65445c9609fb40b2e52828
I20260810 04:49:45.106617 12982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:45.123232 12982 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:45.123976 12982 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:45.128705 12982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6: Generated new TSK 0
I20260810 04:49:45.128903 12982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:49:45.130607 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44573
--builtin_ntp_servers=127.14.237.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:45.219851 13012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:45.220052 13012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:45.220084 13012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:45.221156 13012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:45.221223 13012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:45.221313 13012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:49:45.222645 13012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:45.222896 13012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:45.223191 13012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:45.228176 13025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:45.228189 13024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:45.228259 13029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:45.228479 13012 server_base.cc:1034] running on GCE node
I20260810 04:49:45.228708 13012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:45.228942 13012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:45.230122 13012 hybrid_clock.cc:648] HybridClock initialized: now 1786337385230104 us; error 35 us; skew 500 ppm
I20260810 04:49:45.231392 13012 webserver.cc:460] Webserver started at http://127.14.237.1:41319/ using document root <none> and password file <none>
I20260810 04:49:45.231657 13012 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:45.231802 13012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:45.231963 13012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:45.232990 13012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "7cf8d07cf37240c2823fa8448ff052ec"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "6965b4b95cb43691cf33c4644f649d5d"
server_key_iv: "c71caa246a87f89adebd11d6131e9901"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.233405 13012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "7cf8d07cf37240c2823fa8448ff052ec"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "6965b4b95cb43691cf33c4644f649d5d"
server_key_iv: "c71caa246a87f89adebd11d6131e9901"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.234697 13012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:45.235430 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.235663 13012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:45.235751 13012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "7cf8d07cf37240c2823fa8448ff052ec"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "6965b4b95cb43691cf33c4644f649d5d"
server_key_iv: "c71caa246a87f89adebd11d6131e9901"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.235816 13012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:45.258317 13012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:45.258579 13012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:45.258678 13012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:45.258852 13012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:45.259203 13012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:45.259240 13012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.259262 13012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:45.259310 13012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.265069 13012 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:41315
I20260810 04:49:45.265170 13216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:41315 every 8 connection(s)
I20260810 04:49:45.265494 13012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:49:45.267023 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 13012
I20260810 04:49:45.267129 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:49:45.267372 15284 external_mini_cluster.cc:1428] Setting key 434f9e93769e1cbbe519ee4e654eb777
I20260810 04:49:45.268366 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44573
--builtin_ntp_servers=127.14.237.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:49:45.271798 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.266305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:38517 (local address 127.14.237.62:44573)
0810 04:49:45.266365 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.266367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.266583 (+ 216us) server_negotiation.cc:407] Connection header received
0810 04:49:45.267359 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.267360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.267371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.267454 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:45.268370 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.268922 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.270327 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.270623 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.270701 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.271507 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:45.271509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:45.271509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:45.271514 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:45.271531 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:45.271536 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:45.271593 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.271658 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.271683 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:49:45.271971 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.266305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:44573 (local address 127.14.237.1:38517)
0810 04:49:45.266469 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.266502 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:45.267202 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:45.267533 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:45.267538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:45.267548 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:45.268277 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.268286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.269015 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.269016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.270247 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.270250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.270713 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.270713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.270822 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.271370 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:45.271370 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:45.271397 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:45.271417 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:45.271600 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:45.271606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:45.271609 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:45.271659 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:49:45.271735 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260810 04:49:45.272470 13217 heartbeater.cc:347] Connected to a master server at 127.14.237.62:44573
I20260810 04:49:45.272570 13217 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:45.272791 13217 heartbeater.cc:511] Master 127.14.237.62:44573 requested a full tablet report, sending...
I20260810 04:49:45.273396 12912 ts_manager.cc:194] Registered new tserver with Master: 7cf8d07cf37240c2823fa8448ff052ec (127.14.237.1:41315)
I20260810 04:49:45.274004 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.1:38517
W20260810 04:49:45.353277 13221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:45.353478 13221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:45.353500 13221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:45.354700 13221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:45.354739 13221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:45.354765 13221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:49:45.356076 13221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:45.356351 13221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:45.356658 13221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:45.361626 13242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:45.361698 13235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:45.361637 13234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:45.362004 13221 server_base.cc:1034] running on GCE node
I20260810 04:49:45.362291 13221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:45.362541 13221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:45.363682 13221 hybrid_clock.cc:648] HybridClock initialized: now 1786337385363656 us; error 41 us; skew 500 ppm
I20260810 04:49:45.364871 13221 webserver.cc:460] Webserver started at http://127.14.237.2:41315/ using document root <none> and password file <none>
I20260810 04:49:45.365170 13221 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:45.365291 13221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:45.365461 13221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:45.366509 13221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "8b486d2a90c44c699030b7cc5d80d52f"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "7fafa627fc0a8c65c0754527f13c244f"
server_key_iv: "e19a2f05e867a2fc85f8702d6554f159"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.366899 13221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "8b486d2a90c44c699030b7cc5d80d52f"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "7fafa627fc0a8c65c0754527f13c244f"
server_key_iv: "e19a2f05e867a2fc85f8702d6554f159"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.368182 13221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:49:45.368867 13253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.369036 13221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:45.369149 13221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "8b486d2a90c44c699030b7cc5d80d52f"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "7fafa627fc0a8c65c0754527f13c244f"
server_key_iv: "e19a2f05e867a2fc85f8702d6554f159"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.369251 13221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:45.386241 13221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:45.386538 13221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:45.386714 13221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:45.386921 13221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:45.387233 13221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:45.387285 13221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.387326 13221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:45.387353 13221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.392750 13221 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:34255
I20260810 04:49:45.392830 13439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:34255 every 8 connection(s)
I20260810 04:49:45.393154 13221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:49:45.394812 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 13221
I20260810 04:49:45.394956 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:49:45.395174 15284 external_mini_cluster.cc:1428] Setting key 55858c0dd620a64fea5f6f0ddb160e65
I20260810 04:49:45.396324 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44573
--builtin_ntp_servers=127.14.237.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:49:45.399178 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.393937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:57523 (local address 127.14.237.62:44573)
0810 04:49:45.393998 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.394000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.394286 (+ 286us) server_negotiation.cc:407] Connection header received
0810 04:49:45.394899 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.394901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.394933 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.395077 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:45.395979 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.396681 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.397657 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.397953 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.398034 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.398887 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:45.398888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:45.398889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:45.398893 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:45.398907 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:45.398911 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:45.398976 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.399050 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.399074 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:49:45.399362 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.393926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:44573 (local address 127.14.237.2:57523)
0810 04:49:45.394165 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.394208 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:45.394699 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:45.395139 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:45.395143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:45.395154 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:45.395893 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.395902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.396774 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.396775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.397564 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.397566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.398109 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.398110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.398222 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.398745 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:45.398745 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:45.398771 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:45.398790 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:45.398976 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:45.398982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:45.398985 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:45.399051 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:49:45.399146 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260810 04:49:45.399788 13440 heartbeater.cc:347] Connected to a master server at 127.14.237.62:44573
I20260810 04:49:45.399889 13440 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:45.400105 13440 heartbeater.cc:511] Master 127.14.237.62:44573 requested a full tablet report, sending...
I20260810 04:49:45.400528 12912 ts_manager.cc:194] Registered new tserver with Master: 8b486d2a90c44c699030b7cc5d80d52f (127.14.237.2:34255)
I20260810 04:49:45.400899 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.2:57523
W20260810 04:49:45.477702 13447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:45.477929 13447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:45.477964 13447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:45.479043 13447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:45.479081 13447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:45.479104 13447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:49:45.480460 13447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:45.480691 13447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:45.481012 13447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:45.485947 13454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:45.485962 13455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:45.485962 13458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:45.486232 13447 server_base.cc:1034] running on GCE node
I20260810 04:49:45.486438 13447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:45.486624 13447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:45.487757 13447 hybrid_clock.cc:648] HybridClock initialized: now 1786337385487738 us; error 36 us; skew 500 ppm
I20260810 04:49:45.489030 13447 webserver.cc:460] Webserver started at http://127.14.237.3:42587/ using document root <none> and password file <none>
I20260810 04:49:45.489334 13447 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:45.489385 13447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:45.489531 13447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:45.490581 13447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "4e6418178c194c9ca81ca3b210a7ca06"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "5eeb4b73942b656660183bf7a2cf0d37"
server_key_iv: "f6d50d273f3f97c4e695f863afcd83e5"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.490947 13447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "4e6418178c194c9ca81ca3b210a7ca06"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "5eeb4b73942b656660183bf7a2cf0d37"
server_key_iv: "f6d50d273f3f97c4e695f863afcd83e5"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.492220 13447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:49:45.492966 13471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.493180 13447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:45.493320 13447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "4e6418178c194c9ca81ca3b210a7ca06"
format_stamp: "Formatted at 2026-08-10 04:49:45 on dist-test-slave-8hhm"
server_key: "5eeb4b73942b656660183bf7a2cf0d37"
server_key_iv: "f6d50d273f3f97c4e695f863afcd83e5"
server_key_version: "encryptionkey@0"
I20260810 04:49:45.493427 13447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:45.501011 13447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:45.501332 13447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:45.501471 13447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:45.501706 13447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:45.502077 13447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:45.502111 13447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.502131 13447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:45.502171 13447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:45.507745 13447 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:45907
I20260810 04:49:45.507812 13641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:45907 every 8 connection(s)
I20260810 04:49:45.508123 13447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:49:45.511981 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 13447
I20260810 04:49:45.512084 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:49:45.512302 15284 external_mini_cluster.cc:1428] Setting key 74c16159be014f4c4a3211dd88e5271d
I20260810 04:49:45.513200 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.508807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:39307 (local address 127.14.237.62:44573)
0810 04:49:45.508884 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.508886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.509112 (+ 226us) server_negotiation.cc:407] Connection header received
0810 04:49:45.509650 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.509651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.509663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.509710 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:45.510345 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.510823 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.511706 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.511954 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.512029 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.512941 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:45.512943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:45.512943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:45.512948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:45.512964 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:45.512969 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:45.513021 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.513101 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.513125 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 04:49:45.513465 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.508804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:44573 (local address 127.14.237.3:39307)
0810 04:49:45.509000 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.509032 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:45.509543 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:45.509770 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:45.509774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:45.509787 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:45.510241 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.510245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.510914 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.510915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.511606 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.511608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.512036 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.512037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.512208 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.512784 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:45.512785 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:45.512812 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:45.512833 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:45.513034 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:45.513039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:45.513042 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:45.513105 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:49:45.513199 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260810 04:49:45.513919 13642 heartbeater.cc:347] Connected to a master server at 127.14.237.62:44573
I20260810 04:49:45.514007 13642 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:45.514190 13642 heartbeater.cc:511] Master 127.14.237.62:44573 requested a full tablet report, sending...
I20260810 04:49:45.514554 12912 ts_manager.cc:194] Registered new tserver with Master: 4e6418178c194c9ca81ca3b210a7ca06 (127.14.237.3:45907)
I20260810 04:49:45.514871 12912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.3:39307
I20260810 04:49:45.524159 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:49:45.528167 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.525608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.14.237.62:44573)
0810 04:49:45.525666 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.525668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.525783 (+ 115us) server_negotiation.cc:407] Connection header received
0810 04:49:45.525822 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.525823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.525832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.525881 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:45.526021 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.526489 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.527121 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.527302 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.527349 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.527982 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:45.527983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:45.527983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:45.527987 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:45.527996 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:45.527999 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:45.528035 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.528085 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.528116 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:49:45.529625 12912 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:49:45.531070 12912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:49:45.537256 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.533271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41315 (local address 127.0.0.1:44184)
0810 04:49:45.533468 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.533528 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:45.533592 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:45.534047 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:45.534047 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:45.534050 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:45.534105 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.534107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.535021 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.535022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.536361 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.536363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.536998 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.536999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.537040 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.537043 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:45.537069 (+ 26us) client_negotiation.cc:240] Negotiation successful
0810 04:49:45.537118 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260810 04:49:45.537266 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.533647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.14.237.3:45907)
0810 04:49:45.533819 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.533822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.533830 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:49:45.533858 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.533860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.533871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.533963 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:45.534131 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.535134 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.536217 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.536840 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.536913 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.536993 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.537118 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.537167 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260810 04:49:45.537461 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.533274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:45907 (local address 127.0.0.1:49206)
0810 04:49:45.533641 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.533651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:45.533708 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:45.533985 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:45.533988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:45.533990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:45.534052 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.534054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.535215 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.535216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.536157 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.536160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.536929 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.536931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.537009 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.537023 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:49:45.537091 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:49:45.537120 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":131,"threads_started":1}
I20260810 04:49:45.537559 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.533614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.14.237.2:34255)
0810 04:49:45.533770 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.533775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.534172 (+ 397us) server_negotiation.cc:407] Connection header received
0810 04:49:45.534214 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.534216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.534230 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.534300 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:45.534386 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.535313 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.536167 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.536840 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.536913 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.536993 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.537444 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.537482 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260810 04:49:45.537750 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.533507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:34255 (local address 127.0.0.1:46742)
0810 04:49:45.534042 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.534051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:45.534097 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:45.534299 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:45.534299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:45.534301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:45.534331 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.534332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.535382 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.535383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.536060 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.536062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.537150 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.537151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.537204 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.537228 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 04:49:45.537280 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:49:45.537690 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":7,"thread_start_us":74,"threads_started":1}
I20260810 04:49:45.537786 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.533354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.14.237.1:41315)
0810 04:49:45.533838 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.533844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.533857 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:49:45.533889 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.533891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.533902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.534182 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:45.534247 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.534918 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.536434 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.536908 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.537206 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.537322 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.537671 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.537707 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":119,"threads_started":1}
I20260810 04:49:45.538214 13551 tablet_service.cc:1467] Processing CreateTablet for tablet d56b38206dcb4410ab6d63a13e41d41d (DEFAULT_TABLE table=test-table [id=8882312e02c74ed484529d74858e142c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:45.538458 13110 tablet_service.cc:1467] Processing CreateTablet for tablet d56b38206dcb4410ab6d63a13e41d41d (DEFAULT_TABLE table=test-table [id=8882312e02c74ed484529d74858e142c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:45.538658 13551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56b38206dcb4410ab6d63a13e41d41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:45.538828 13110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56b38206dcb4410ab6d63a13e41d41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:45.541383 13665 tablet_bootstrap.cc:492] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Bootstrap starting.
I20260810 04:49:45.541712 13669 tablet_bootstrap.cc:492] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec: Bootstrap starting.
I20260810 04:49:45.541888 13326 tablet_service.cc:1467] Processing CreateTablet for tablet d56b38206dcb4410ab6d63a13e41d41d (DEFAULT_TABLE table=test-table [id=8882312e02c74ed484529d74858e142c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:45.542065 13665 tablet_bootstrap.cc:654] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:45.542243 13326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56b38206dcb4410ab6d63a13e41d41d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:45.542376 13669 tablet_bootstrap.cc:654] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:45.542424 13665 log.cc:824] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:45.542706 13669 log.cc:824] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:45.543198 13665 tablet_bootstrap.cc:492] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: No bootstrap required, opened a new log
I20260810 04:49:45.543344 13665 ts_tablet_manager.cc:1397] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:45.543478 13669 tablet_bootstrap.cc:492] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec: No bootstrap required, opened a new log
I20260810 04:49:45.543546 13669 ts_tablet_manager.cc:1397] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:49:45.545662 13673 tablet_bootstrap.cc:492] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Bootstrap starting.
I20260810 04:49:45.545837 13669 raft_consensus.cc:348] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.546017 13669 raft_consensus.cc:374] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:45.545920 13665 raft_consensus.cc:348] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.546059 13669 raft_consensus.cc:729] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf8d07cf37240c2823fa8448ff052ec, State: Initialized, Role: FOLLOWER
I20260810 04:49:45.546067 13665 raft_consensus.cc:374] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:45.546100 13665 raft_consensus.cc:729] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6418178c194c9ca81ca3b210a7ca06, State: Initialized, Role: FOLLOWER
I20260810 04:49:45.546206 13669 consensus_queue.cc:260] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.546231 13665 consensus_queue.cc:260] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.546346 13673 tablet_bootstrap.cc:654] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:45.546494 13665 ts_tablet_manager.cc:1428] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:49:45.546502 13669 ts_tablet_manager.cc:1428] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:49:45.546631 13642 heartbeater.cc:503] Master 127.14.237.62:44573 was elected leader, sending a full tablet report...
I20260810 04:49:45.546644 13217 heartbeater.cc:503] Master 127.14.237.62:44573 was elected leader, sending a full tablet report...
I20260810 04:49:45.546672 13673 log.cc:824] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:45.547331 13673 tablet_bootstrap.cc:492] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: No bootstrap required, opened a new log
I20260810 04:49:45.547560 13673 ts_tablet_manager.cc:1397] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:45.549448 13673 raft_consensus.cc:348] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.549590 13673 raft_consensus.cc:374] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:45.549635 13673 raft_consensus.cc:729] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b486d2a90c44c699030b7cc5d80d52f, State: Initialized, Role: FOLLOWER
I20260810 04:49:45.549742 13673 consensus_queue.cc:260] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.549994 13673 ts_tablet_manager.cc:1428] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:45.550071 13440 heartbeater.cc:503] Master 127.14.237.62:44573 was elected leader, sending a full tablet report...
W20260810 04:49:45.643739 13442 tablet.cc:2369] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:49:45.745452 13677 raft_consensus.cc:482] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:45.745625 13677 raft_consensus.cc:504] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.746004 13677 leader_election.cc:288] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cf8d07cf37240c2823fa8448ff052ec (127.14.237.1:41315), 4e6418178c194c9ca81ca3b210a7ca06 (127.14.237.3:45907)
I20260810 04:49:45.750037 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.746296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:56879 (local address 127.14.237.3:45907)
0810 04:49:45.746336 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.746337 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.746432 (+ 95us) server_negotiation.cc:407] Connection header received
0810 04:49:45.746468 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.746469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.746478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.746543 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:45.746798 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.747529 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.749042 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.749620 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.749691 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.749699 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.749904 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.749968 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:49:45.750037 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.746242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:50975 (local address 127.14.237.1:41315)
0810 04:49:45.746300 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.746303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.746314 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:49:45.746422 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.746423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.746437 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.746520 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:45.746804 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.747536 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.749048 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.749609 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.749691 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.749699 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.749904 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.749967 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:49:45.750033 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.746177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41315 (local address 127.14.237.2:50975)
0810 04:49:45.746251 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.746274 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:45.746317 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:45.746560 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:45.746560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:45.746562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:45.746684 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.746686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.747628 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.747628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.748934 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.748937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.749714 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.749715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.749791 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.749796 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:49:45.749859 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:49:45.749907 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":141}
I20260810 04:49:45.750041 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.746177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:45907 (local address 127.14.237.2:56879)
0810 04:49:45.746347 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.746355 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:45.746425 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:45.746560 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:45.746560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:45.746562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:45.746684 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.746686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.747628 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.747628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.748934 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:45.748937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.749715 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.749716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:45.749805 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.749807 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:49:45.749859 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:49:45.749905 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":80,"thread_start_us":91,"threads_started":1}
I20260810 04:49:45.750407 13576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56b38206dcb4410ab6d63a13e41d41d" candidate_uuid: "8b486d2a90c44c699030b7cc5d80d52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6418178c194c9ca81ca3b210a7ca06" is_pre_election: true
I20260810 04:49:45.750401 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56b38206dcb4410ab6d63a13e41d41d" candidate_uuid: "8b486d2a90c44c699030b7cc5d80d52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf8d07cf37240c2823fa8448ff052ec" is_pre_election: true
I20260810 04:49:45.750557 13147 raft_consensus.cc:2454] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b486d2a90c44c699030b7cc5d80d52f in term 0.
I20260810 04:49:45.750557 13576 raft_consensus.cc:2454] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b486d2a90c44c699030b7cc5d80d52f in term 0.
I20260810 04:49:45.750808 13256 leader_election.cc:302] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf8d07cf37240c2823fa8448ff052ec, 8b486d2a90c44c699030b7cc5d80d52f; no voters:
I20260810 04:49:45.750932 13677 raft_consensus.cc:2781] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:49:45.750988 13677 raft_consensus.cc:482] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:49:45.751020 13677 raft_consensus.cc:3037] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:45.751706 13677 raft_consensus.cc:504] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.751864 13677 leader_election.cc:288] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [CANDIDATE]: Term 1 election: Requested vote from peers 7cf8d07cf37240c2823fa8448ff052ec (127.14.237.1:41315), 4e6418178c194c9ca81ca3b210a7ca06 (127.14.237.3:45907)
I20260810 04:49:45.752008 13576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56b38206dcb4410ab6d63a13e41d41d" candidate_uuid: "8b486d2a90c44c699030b7cc5d80d52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6418178c194c9ca81ca3b210a7ca06"
I20260810 04:49:45.752053 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56b38206dcb4410ab6d63a13e41d41d" candidate_uuid: "8b486d2a90c44c699030b7cc5d80d52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf8d07cf37240c2823fa8448ff052ec"
I20260810 04:49:45.752105 13576 raft_consensus.cc:3037] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:45.752168 13147 raft_consensus.cc:3037] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:45.752849 13576 raft_consensus.cc:2454] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b486d2a90c44c699030b7cc5d80d52f in term 1.
I20260810 04:49:45.752976 13147 raft_consensus.cc:2454] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b486d2a90c44c699030b7cc5d80d52f in term 1.
I20260810 04:49:45.753098 13259 leader_election.cc:302] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6418178c194c9ca81ca3b210a7ca06, 8b486d2a90c44c699030b7cc5d80d52f; no voters:
I20260810 04:49:45.753189 13677 raft_consensus.cc:2781] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:45.753371 13677 raft_consensus.cc:686] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 1 LEADER]: Becoming Leader. State: Replica: 8b486d2a90c44c699030b7cc5d80d52f, State: Running, Role: LEADER
I20260810 04:49:45.753504 13677 consensus_queue.cc:237] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } }
I20260810 04:49:45.754386 12912 catalog_manager.cc:5495] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b486d2a90c44c699030b7cc5d80d52f (127.14.237.2). New cstate: current_term: 1 leader_uuid: "8b486d2a90c44c699030b7cc5d80d52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b486d2a90c44c699030b7cc5d80d52f" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 34255 } health_report { overall_health: HEALTHY } } }
W20260810 04:49:45.758642 13643 tablet.cc:2369] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:49:45.762024 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.759968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.14.237.2:34255)
0810 04:49:45.760005 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:45.760007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:45.760049 (+ 42us) server_negotiation.cc:407] Connection header received
0810 04:49:45.760107 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:45.760107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:45.760121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:45.760178 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:49:45.760272 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.760671 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.761435 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:45.761603 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:45.761655 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:45.761784 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:49:45.761865 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:49:45.761919 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:45.761950 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 04:49:45.761963 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:49:45.763691 13147 raft_consensus.cc:1261] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec [term 1 FOLLOWER]: Refusing update from remote peer 8b486d2a90c44c699030b7cc5d80d52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:45.763693 13576 raft_consensus.cc:1261] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 1 FOLLOWER]: Refusing update from remote peer 8b486d2a90c44c699030b7cc5d80d52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:45.764034 13679 consensus_queue.cc:1035] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6418178c194c9ca81ca3b210a7ca06" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 45907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:45.764149 13677 consensus_queue.cc:1035] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cf8d07cf37240c2823fa8448ff052ec" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 04:49:45.766137 13218 tablet.cc:2369] T d56b38206dcb4410ab6d63a13e41d41d P 7cf8d07cf37240c2823fa8448ff052ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:49:45.766603 13684 mvcc.cc:205] Tried to move back new op lower bound from 7316837932085338112 to 7316837932046946304. Current Snapshot: MvccSnapshot[applied={T|T < 7316837932085338112}]
I20260810 04:49:45.766997 13683 mvcc.cc:205] Tried to move back new op lower bound from 7316837932085338112 to 7316837932046946304. Current Snapshot: MvccSnapshot[applied={T|T < 7316837932085338112}]
I20260810 04:49:45.767230 13686 mvcc.cc:205] Tried to move back new op lower bound from 7316837932085338112 to 7316837932046946304. Current Snapshot: MvccSnapshot[applied={T|T < 7316837932085338112 or (T in {7316837932085338112})}]
I20260810 04:49:45.768929 12910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:45.769014 12910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:45.769711 12910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c4e4d7d55c342caba76191771e9f7c6: Sending DeleteTablet for 3 replicas of tablet d56b38206dcb4410ab6d63a13e41d41d
I20260810 04:49:45.770069 13326 tablet_service.cc:1514] Processing DeleteTablet for tablet d56b38206dcb4410ab6d63a13e41d41d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:45 UTC) from {username='slave'} at 127.0.0.1:46742
I20260810 04:49:45.770071 13110 tablet_service.cc:1514] Processing DeleteTablet for tablet d56b38206dcb4410ab6d63a13e41d41d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:45 UTC) from {username='slave'} at 127.0.0.1:44184
I20260810 04:49:45.770078 13551 tablet_service.cc:1514] Processing DeleteTablet for tablet d56b38206dcb4410ab6d63a13e41d41d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:45 UTC) from {username='slave'} at 127.0.0.1:49206
I20260810 04:49:45.770427 13707 tablet_replica.cc:330] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: stopping tablet replica
I20260810 04:49:45.770561 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 13012
I20260810 04:49:45.778366 13707 raft_consensus.cc:2229] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:45.778512 13707 raft_consensus.cc:2258] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:45.779675 13704 tablet_replica.cc:330] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: stopping tablet replica
I20260810 04:49:45.779815 13704 raft_consensus.cc:2229] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:49:45.779947 13704 raft_consensus.cc:2258] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:45.779950 13707 ts_tablet_manager.cc:1905] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:45.780404 13704 ts_tablet_manager.cc:1905] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:45.782191 13707 ts_tablet_manager.cc:1918] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:45.782196 13704 ts_tablet_manager.cc:1918] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:45.782289 13707 log.cc:1196] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/d56b38206dcb4410ab6d63a13e41d41d
I20260810 04:49:45.782337 13704 log.cc:1196] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/d56b38206dcb4410ab6d63a13e41d41d
I20260810 04:49:45.782562 13707 ts_tablet_manager.cc:1939] T d56b38206dcb4410ab6d63a13e41d41d P 4e6418178c194c9ca81ca3b210a7ca06: Deleting consensus metadata
I20260810 04:49:45.782706 13704 ts_tablet_manager.cc:1939] T d56b38206dcb4410ab6d63a13e41d41d P 8b486d2a90c44c699030b7cc5d80d52f: Deleting consensus metadata
I20260810 04:49:45.782982 12891 catalog_manager.cc:4841] TS 4e6418178c194c9ca81ca3b210a7ca06 (127.14.237.3:45907): tablet d56b38206dcb4410ab6d63a13e41d41d (table test-table [id=8882312e02c74ed484529d74858e142c]) successfully deleted
I20260810 04:49:45.783120 12885 catalog_manager.cc:4841] TS 8b486d2a90c44c699030b7cc5d80d52f (127.14.237.2:34255): tablet d56b38206dcb4410ab6d63a13e41d41d (table test-table [id=8882312e02c74ed484529d74858e142c]) successfully deleted
W20260810 04:49:45.786926 12885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:41315 (error 108)
I20260810 04:49:45.787446 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:45.787293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:41315 (local address 127.0.0.1:44196)
0810 04:49:45.787364 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:45.787410 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:41315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 04:49:45.787554 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 13221
W20260810 04:49:45.787562 12885 catalog_manager.cc:4584] TS 7cf8d07cf37240c2823fa8448ff052ec (127.14.237.1:41315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d56b38206dcb4410ab6d63a13e41d41d: Network error: Client connection negotiation failed: client connection to 127.14.237.1:41315: connect: Connection refused (error 111)
I20260810 04:49:45.806200 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 13447
I20260810 04:49:45.817339 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 12864
2026-08-10T04:49:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (882 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:45 dist-test-slave-8hhm krb5kdc[13715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:45 dist-test-slave-8hhm krb5kdc[13715](info): set up 2 sockets
Aug 10 04:49:45 dist-test-slave-8hhm krb5kdc[13715](info): commencing operation
krb5kdc: starting...
W20260810 04:49:47.073330 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:49:47.192299 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:43989
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:43989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:47.274022 13736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:47.274248 13736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:47.274288 13736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:47.275398 13736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:47.275462 13736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:47.275485 13736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:47.275506 13736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:47.275533 13736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:47.276983 13736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:43989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:47.277278 13736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:47.277516 13736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:47.282377 13753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:47.282342 13747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:47.282435 13748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:47.282523 13736 server_base.cc:1034] running on GCE node
I20260810 04:49:47.282927 13736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:47.283279 13736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:47.284466 13736 hybrid_clock.cc:648] HybridClock initialized: now 1786337387284449 us; error 25 us; skew 500 ppm
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:47.286950 13736 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:49:47.287317 13736 webserver.cc:460] Webserver started at http://127.14.237.62:36893/ using document root <none> and password file <none>
I20260810 04:49:47.287508 13736 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:47.287546 13736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:47.287631 13736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:47.288550 13736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "19d304bdcbc84f29b723d759ddaf9360"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "cd86ad03e5928d811ff2dc565ec92849"
server_key_iv: "b913dd04e55ad300cf80999e388cd881"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.288846 13736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "19d304bdcbc84f29b723d759ddaf9360"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "cd86ad03e5928d811ff2dc565ec92849"
server_key_iv: "b913dd04e55ad300cf80999e388cd881"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.290081 13736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:49:47.290711 13764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.290886 13736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:47.290975 13736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "19d304bdcbc84f29b723d759ddaf9360"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "cd86ad03e5928d811ff2dc565ec92849"
server_key_iv: "b913dd04e55ad300cf80999e388cd881"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.291044 13736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:47.319628 13736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:47.322026 13736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:47.322186 13736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:47.326268 13736 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:43989
I20260810 04:49:47.326337 13840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:43989 every 8 connection(s)
I20260810 04:49:47.326674 13736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:47.327265 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 13736
I20260810 04:49:47.327358 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:47.327579 15284 external_mini_cluster.cc:1428] Setting key e7ac8729cfb8a7ab35d8f67c74e30263
I20260810 04:49:47.327770 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:47.330717 13841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360: Bootstrap starting.
I20260810 04:49:47.331358 13841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:47.331688 13841 log.cc:824] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:47.332477 13841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360: No bootstrap required, opened a new log
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:47.334946 13841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d304bdcbc84f29b723d759ddaf9360" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43989 } }
I20260810 04:49:47.335129 13841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:47.335171 13841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d304bdcbc84f29b723d759ddaf9360, State: Initialized, Role: FOLLOWER
I20260810 04:49:47.335315 13841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d304bdcbc84f29b723d759ddaf9360" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43989 } }
I20260810 04:49:47.335381 13841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:47.335400 13841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:47.335476 13841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:47.336144 13841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d304bdcbc84f29b723d759ddaf9360" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43989 } }
I20260810 04:49:47.336248 13841 leader_election.cc:302] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d304bdcbc84f29b723d759ddaf9360; no voters:
I20260810 04:49:47.336452 13841 leader_election.cc:288] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:47.336480 13853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:47.336586 13853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [term 1 LEADER]: Becoming Leader. State: Replica: 19d304bdcbc84f29b723d759ddaf9360, State: Running, Role: LEADER
I20260810 04:49:47.336690 13853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d304bdcbc84f29b723d759ddaf9360" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43989 } }
I20260810 04:49:47.336741 13841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:47.337052 13853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d304bdcbc84f29b723d759ddaf9360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d304bdcbc84f29b723d759ddaf9360" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43989 } } }
I20260810 04:49:47.337268 13853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:47.337474 13853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d304bdcbc84f29b723d759ddaf9360. Latest consensus state: current_term: 1 leader_uuid: "19d304bdcbc84f29b723d759ddaf9360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d304bdcbc84f29b723d759ddaf9360" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43989 } } }
I20260810 04:49:47.337549 13853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:47.337940 13872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:47.338572 13872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:47.338946 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.328124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.14.237.62:43989)
0810 04:49:47.328332 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.328340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.328365 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:49:47.328892 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.328898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.328956 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.329100 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:47.329800 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.330643 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.331462 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.331935 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.332055 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.334710 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:47.334714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:47.334714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:47.334733 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:47.336794 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.337288 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.337291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.337291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.337403 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.337637 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.337638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.337638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.337968 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:47.338151 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.338683 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.338808 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
I20260810 04:49:47.340162 13872 catalog_manager.cc:1347] Generated new cluster ID: 74946ad43a514916b6d63553abe73588
I20260810 04:49:47.340219 13872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:47.349884 13872 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:47.350669 13872 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:47.354457 13872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360: Generated new TSK 0
I20260810 04:49:47.354616 13872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:49:47.380056 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43989
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:47.462713 13891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:47.462952 13891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:47.462996 13891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:47.464139 13891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:47.464211 13891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:47.464257 13891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:49:47.465708 13891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:47.465986 13891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:47.466221 13891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:47.471410 13903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:47.471457 13891 server_base.cc:1034] running on GCE node
W20260810 04:49:47.471421 13905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:47.471426 13902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:47.471897 13891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:47.472159 13891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:47.473290 13891 hybrid_clock.cc:648] HybridClock initialized: now 1786337387473281 us; error 39 us; skew 500 ppm
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:47.475910 13891 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:49:47.476338 13891 webserver.cc:460] Webserver started at http://127.14.237.1:39457/ using document root <none> and password file <none>
I20260810 04:49:47.476552 13891 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:47.476619 13891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:47.476769 13891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:47.477844 13891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "866cf46374b44bceacb9a5411f911e30"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "fa4ffc6c5f235228d64bae8b714346af"
server_key_iv: "95fb6784a2a61970b59682e9e4956b38"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.478205 13891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "866cf46374b44bceacb9a5411f911e30"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "fa4ffc6c5f235228d64bae8b714346af"
server_key_iv: "95fb6784a2a61970b59682e9e4956b38"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.479480 13891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:49:47.480139 13915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.480309 13891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:47.480443 13891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "866cf46374b44bceacb9a5411f911e30"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "fa4ffc6c5f235228d64bae8b714346af"
server_key_iv: "95fb6784a2a61970b59682e9e4956b38"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.480511 13891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:47.515831 13891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:47.517576 13891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:47.517735 13891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:47.517971 13891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:47.518265 13891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:47.518316 13891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.518347 13891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:47.518373 13891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.523901 13891 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:42003
I20260810 04:49:47.523952 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:42003 every 8 connection(s)
I20260810 04:49:47.524240 13891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:49:47.525316 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 13891
I20260810 04:49:47.525449 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:49:47.525672 15284 external_mini_cluster.cc:1428] Setting key d065d64675097802fc6184a15b696c85
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:47.534982 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.524948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:40801 (local address 127.14.237.62:43989)
0810 04:49:47.525031 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.525033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.525302 (+ 269us) server_negotiation.cc:407] Connection header received
0810 04:49:47.525904 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.525905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.525916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.526003 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:47.527071 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.527831 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.529260 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.529462 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.529518 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.531960 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:47.531962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:47.531962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:47.531970 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:47.533638 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.534030 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.534031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.534031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.534099 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.534331 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.534332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.534332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.534476 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:47.534554 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.534792 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.534883 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 04:49:47.535000 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.524934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43989 (local address 127.14.237.1:40801)
0810 04:49:47.525154 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:47.525196 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:47.525793 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:47.526087 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:47.526090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:47.526783 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:47.526958 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.526963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.527941 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.527941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.529134 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.529137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.529552 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.529553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.529719 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.530190 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:47.530200 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:47.531883 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:47.533719 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:47.533725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:47.533727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:47.533940 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:47.534167 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:47.534168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:47.534168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:47.534263 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:47.534547 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:47.534551 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:47.534663 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 04:49:47.534740 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:49:47.534849 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":119,"threads_started":1}
I20260810 04:49:47.535540 14083 heartbeater.cc:347] Connected to a master server at 127.14.237.62:43989
I20260810 04:49:47.535651 14083 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:47.535837 14083 heartbeater.cc:511] Master 127.14.237.62:43989 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
I20260810 04:49:47.536439 13787 ts_manager.cc:194] Registered new tserver with Master: 866cf46374b44bceacb9a5411f911e30 (127.14.237.1:42003)
I20260810 04:49:47.537089 13787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:40801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:49:47.554260 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43989
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:47.632918 14091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:47.633111 14091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:47.633136 14091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:47.634227 14091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:47.634268 14091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:47.634292 14091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:49:47.635528 14091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:47.635700 14091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:47.635855 14091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:47.640738 14099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:47.640745 14097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:47.640745 14102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:47.641242 14091 server_base.cc:1034] running on GCE node
I20260810 04:49:47.641469 14091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:47.641755 14091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:47.642999 14091 hybrid_clock.cc:648] HybridClock initialized: now 1786337387642975 us; error 31 us; skew 500 ppm
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:47.650610 14091 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:49:47.651036 14091 webserver.cc:460] Webserver started at http://127.14.237.2:37255/ using document root <none> and password file <none>
I20260810 04:49:47.651250 14091 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:47.651295 14091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:47.651389 14091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:47.652673 14091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "b944db326faa49e2a7e1d4f07c369da3"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "57e42e055bf41baba67e0b83ad4a4bf8"
server_key_iv: "3989ced9a12a78ab15a698e19baf8042"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.653148 14091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "b944db326faa49e2a7e1d4f07c369da3"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "57e42e055bf41baba67e0b83ad4a4bf8"
server_key_iv: "3989ced9a12a78ab15a698e19baf8042"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.655323 14091 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 04:49:47.656371 14118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.656574 14091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:47.656688 14091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "b944db326faa49e2a7e1d4f07c369da3"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "57e42e055bf41baba67e0b83ad4a4bf8"
server_key_iv: "3989ced9a12a78ab15a698e19baf8042"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.656775 14091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:47.671185 14091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:47.673316 14091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:47.673498 14091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:47.673743 14091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:47.674063 14091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:47.674139 14091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.674175 14091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:47.674201 14091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.680012 14091 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:41135
I20260810 04:49:47.680048 14303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:41135 every 8 connection(s)
I20260810 04:49:47.680370 14091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:47.688771 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 14091
I20260810 04:49:47.688892 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:49:47.689123 15284 external_mini_cluster.cc:1428] Setting key 7dce042f71de31818c5421a9876061d2
I20260810 04:49:47.689642 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.681090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:54209 (local address 127.14.237.62:43989)
0810 04:49:47.681162 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.681164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.681397 (+ 233us) server_negotiation.cc:407] Connection header received
0810 04:49:47.681908 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.681909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.681923 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.681993 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:47.682960 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.683529 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.684606 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.684812 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.684888 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.686936 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:47.686938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:47.686938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:47.686946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:47.688138 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.688550 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.688551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.688552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.688626 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.688887 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.688888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.688889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.689058 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:47.689119 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.689457 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.689541 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:49:47.689810 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.681080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43989 (local address 127.14.237.2:54209)
0810 04:49:47.681278 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:47.681312 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:47.681803 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:47.682068 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:47.682071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:47.682639 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:47.682819 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.682824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.683634 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.683635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.684513 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.684515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.684902 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.684903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.685025 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.685505 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:47.685513 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:47.686833 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:47.688240 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:47.688248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:47.688250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:47.688470 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:47.688696 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:47.688697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:47.688698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:47.688816 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:47.689126 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:47.689134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:47.689322 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 04:49:47.689432 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 04:49:47.689582 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260810 04:49:47.690326 14304 heartbeater.cc:347] Connected to a master server at 127.14.237.62:43989
I20260810 04:49:47.690418 14304 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:47.690603 14304 heartbeater.cc:511] Master 127.14.237.62:43989 requested a full tablet report, sending...
I20260810 04:49:47.690954 13787 ts_manager.cc:194] Registered new tserver with Master: b944db326faa49e2a7e1d4f07c369da3 (127.14.237.2:41135)
I20260810 04:49:47.691277 13787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:54209
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:49:47.714347 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43989
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:47.794727 14316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:47.794927 14316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:47.794952 14316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:47.796242 14316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:47.796290 14316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:47.796350 14316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:49:47.797930 14316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:47.798182 14316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:47.798413 14316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:47.803653 14326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:47.803661 14329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:47.803686 14327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:47.803676 14316 server_base.cc:1034] running on GCE node
I20260810 04:49:47.804114 14316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:47.804347 14316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:47.805506 14316 hybrid_clock.cc:648] HybridClock initialized: now 1786337387805499 us; error 34 us; skew 500 ppm
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:47.807940 14316 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:49:47.808413 14316 webserver.cc:460] Webserver started at http://127.14.237.3:45993/ using document root <none> and password file <none>
I20260810 04:49:47.808646 14316 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:47.808713 14316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:47.808864 14316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:47.809883 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "67420769bcdd43bea5efa45f1fac4405"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "5869da1877426ad4d801d2fbbaa52362"
server_key_iv: "fb98c7d4865610487f12bb22fb78ba50"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.810240 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "67420769bcdd43bea5efa45f1fac4405"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "5869da1877426ad4d801d2fbbaa52362"
server_key_iv: "fb98c7d4865610487f12bb22fb78ba50"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.811542 14316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:49:47.812280 14337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.812458 14316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:47.812565 14316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "67420769bcdd43bea5efa45f1fac4405"
format_stamp: "Formatted at 2026-08-10 04:49:47 on dist-test-slave-8hhm"
server_key: "5869da1877426ad4d801d2fbbaa52362"
server_key_iv: "fb98c7d4865610487f12bb22fb78ba50"
server_key_version: "encryptionkey@0"
I20260810 04:49:47.812644 14316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:47.826436 14316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:47.828168 14316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:47.828325 14316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:47.828565 14316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:47.828864 14316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:47.828918 14316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.828958 14316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:47.828982 14316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:47.834798 14316 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:34595
I20260810 04:49:47.834843 14527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:34595 every 8 connection(s)
I20260810 04:49:47.835130 14316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:49:47.838829 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 14316
I20260810 04:49:47.838950 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:49:47.839138 15284 external_mini_cluster.cc:1428] Setting key 7243f0325d6840fef22bf8d1908f0948
Aug 10 04:49:47 dist-test-slave-8hhm krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:47.844725 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.835836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:45883 (local address 127.14.237.62:43989)
0810 04:49:47.835925 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.835927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.836079 (+ 152us) server_negotiation.cc:407] Connection header received
0810 04:49:47.836682 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.836684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.836698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.836757 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:47.837907 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.838527 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.839926 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.840194 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.840251 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.842342 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:47.842344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:47.842344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:47.842350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:47.843555 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.843912 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.843913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.843913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.843964 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.844243 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.844244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.844244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.844339 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:47.844386 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.844600 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.844665 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 04:49:47.844866 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.835785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43989 (local address 127.14.237.3:45883)
0810 04:49:47.835956 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:47.835995 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:47.836560 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:47.836823 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:47.836826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:47.837563 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:47.837798 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.837804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.838644 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.838645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.839837 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.839840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.840269 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.840270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.840378 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.840889 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:47.840897 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:47.842265 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:47.843629 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:47.843635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:47.843636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:47.843849 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:47.844034 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:47.844035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:47.844035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:47.844181 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:47.844393 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:47.844398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:47.844498 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 04:49:47.844575 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:49:47.844682 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260810 04:49:47.845402 14529 heartbeater.cc:347] Connected to a master server at 127.14.237.62:43989
I20260810 04:49:47.845487 14529 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:47.845616 14529 heartbeater.cc:511] Master 127.14.237.62:43989 requested a full tablet report, sending...
I20260810 04:49:47.845939 13787 ts_manager.cc:194] Registered new tserver with Master: 67420769bcdd43bea5efa45f1fac4405 (127.14.237.3:34595)
I20260810 04:49:47.846256 13787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:45883
I20260810 04:49:47.850790 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:49:47.858171 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.852087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.14.237.62:43989)
0810 04:49:47.852162 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.852164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.852255 (+ 91us) server_negotiation.cc:407] Connection header received
0810 04:49:47.852319 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.852320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.852332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.852399 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:47.852973 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.853496 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.854210 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.854415 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.854506 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.855713 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:47.855715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:47.855715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:47.855721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:47.857036 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.857505 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.857506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.857507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.857558 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:47.857755 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:47.857756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:47.857756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:47.857864 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:47.857937 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.858035 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.858081 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 04:49:47.859715 13787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:49:47.861397 13787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:49:47.868320 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.863548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:41135 (local address 127.0.0.1:46974)
0810 04:49:47.864036 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:47.864047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:47.864111 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:47.864610 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:47.864611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:47.864613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:47.864663 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.864664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.865629 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.865630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.866994 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.866997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.868005 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.868006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.868091 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.868106 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 04:49:47.868179 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:49:47.868243 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":81,"thread_start_us":105,"threads_started":1}
I20260810 04:49:47.868369 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.863829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.14.237.2:41135)
0810 04:49:47.864038 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.864042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.864146 (+ 104us) server_negotiation.cc:407] Connection header received
0810 04:49:47.864208 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.864210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.864437 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.864515 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:47.864736 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.865519 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.867084 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.867784 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.867908 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.867965 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.868210 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.868266 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
I20260810 04:49:47.868534 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.863712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.14.237.3:34595)
0810 04:49:47.864163 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.864166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.864176 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:49:47.864207 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.864209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.864225 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.864324 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:47.864506 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.865646 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.867084 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.867784 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.867906 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.867965 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.868286 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.868324 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":71,"threads_started":1}
I20260810 04:49:47.868938 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.863523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:34595 (local address 127.0.0.1:50054)
0810 04:49:47.863713 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:47.863744 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:47.863820 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:47.864334 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:47.864335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:47.864337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:47.864415 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.864417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.865756 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.865757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.866994 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.866997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.868005 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.868006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.868091 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.868106 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 04:49:47.868172 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:49:47.868241 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":30,"thread_start_us":93,"threads_started":1}
I20260810 04:49:47.868991 14202 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900 (DEFAULT_TABLE table=test-table [id=073f17cdec7c4f00b76cb09a265072f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:47.869405 14202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:47.869801 14423 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900 (DEFAULT_TABLE table=test-table [id=073f17cdec7c4f00b76cb09a265072f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:47.870117 14423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:47.871560 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.863827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.14.237.1:42003)
0810 04:49:47.864012 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.864016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.864028 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:49:47.864070 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.864072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.864087 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.864181 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:47.864508 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.865650 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.870409 (+ 4759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.871151 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.871234 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.871291 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.871435 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.871486 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260810 04:49:47.871978 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.863523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42003 (local address 127.0.0.1:41356)
0810 04:49:47.863919 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:47.863927 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:47.863960 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:47.864320 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:47.864323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:47.864326 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:47.864415 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.864419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.867095 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.867096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.867751 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.867752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.871258 (+ 3506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.871259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.871330 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.871334 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:47.871405 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:49:47.871874 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":6,"thread_start_us":87,"threads_started":1}
I20260810 04:49:47.872676 13983 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900 (DEFAULT_TABLE table=test-table [id=073f17cdec7c4f00b76cb09a265072f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:47.872860 14564 tablet_bootstrap.cc:492] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: Bootstrap starting.
I20260810 04:49:47.873056 13983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:47.873226 14565 tablet_bootstrap.cc:492] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Bootstrap starting.
I20260810 04:49:47.873513 14564 tablet_bootstrap.cc:654] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:47.873857 14565 tablet_bootstrap.cc:654] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:47.873937 14564 log.cc:824] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:47.874431 14565 log.cc:824] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:47.874573 14564 tablet_bootstrap.cc:492] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: No bootstrap required, opened a new log
I20260810 04:49:47.874616 14564 ts_tablet_manager.cc:1397] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:49:47.875490 14565 tablet_bootstrap.cc:492] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: No bootstrap required, opened a new log
I20260810 04:49:47.875559 14565 ts_tablet_manager.cc:1397] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:47.875947 14564 raft_consensus.cc:348] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.876091 14564 raft_consensus.cc:374] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:47.876111 14564 raft_consensus.cc:729] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67420769bcdd43bea5efa45f1fac4405, State: Initialized, Role: FOLLOWER
I20260810 04:49:47.876226 14564 consensus_queue.cc:260] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.876492 14529 heartbeater.cc:503] Master 127.14.237.62:43989 was elected leader, sending a full tablet report...
I20260810 04:49:47.876497 14564 ts_tablet_manager.cc:1428] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:47.877099 14571 tablet_bootstrap.cc:492] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30: Bootstrap starting.
I20260810 04:49:47.877660 14565 raft_consensus.cc:348] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.877807 14571 tablet_bootstrap.cc:654] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:47.877853 14565 raft_consensus.cc:374] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:47.877893 14565 raft_consensus.cc:729] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b944db326faa49e2a7e1d4f07c369da3, State: Initialized, Role: FOLLOWER
I20260810 04:49:47.878031 14565 consensus_queue.cc:260] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.878362 14565 ts_tablet_manager.cc:1428] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:49:47.878376 14571 log.cc:824] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:47.878365 14304 heartbeater.cc:503] Master 127.14.237.62:43989 was elected leader, sending a full tablet report...
I20260810 04:49:47.879202 14571 tablet_bootstrap.cc:492] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30: No bootstrap required, opened a new log
I20260810 04:49:47.879283 14571 ts_tablet_manager.cc:1397] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:47.880518 14571 raft_consensus.cc:348] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.880646 14571 raft_consensus.cc:374] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:47.880667 14571 raft_consensus.cc:729] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 866cf46374b44bceacb9a5411f911e30, State: Initialized, Role: FOLLOWER
I20260810 04:49:47.880787 14571 consensus_queue.cc:260] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.881011 14571 ts_tablet_manager.cc:1428] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:47.881083 14083 heartbeater.cc:503] Master 127.14.237.62:43989 was elected leader, sending a full tablet report...
W20260810 04:49:47.930989 14306 tablet.cc:2369] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:49:47.936360 14572 raft_consensus.cc:482] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:47.936470 14572 raft_consensus.cc:504] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.936921 14572 leader_election.cc:288] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 866cf46374b44bceacb9a5411f911e30 (127.14.237.1:42003), 67420769bcdd43bea5efa45f1fac4405 (127.14.237.3:34595)
I20260810 04:49:47.940979 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.937258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:43047 (local address 127.14.237.3:34595)
0810 04:49:47.937324 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.937344 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.937390 (+ 46us) server_negotiation.cc:407] Connection header received
0810 04:49:47.937481 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.937482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.937491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.937560 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:47.937743 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.938472 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.939989 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.940577 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.940655 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.940664 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.940848 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.940905 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:49:47.940979 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.937304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:55283 (local address 127.14.237.1:42003)
0810 04:49:47.937377 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.937379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.937389 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:49:47.937418 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.937419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.937428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.937500 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:47.937701 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.938489 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.939989 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.940577 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.940656 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.940664 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.940848 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.940907 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:49:47.940989 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.937055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42003 (local address 127.14.237.2:55283)
0810 04:49:47.937116 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:47.937128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:47.937173 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:47.937542 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:47.937544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:47.937546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:47.937629 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.937631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.938575 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.938576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.939887 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.939891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.940670 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.940671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.940755 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.940762 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 04:49:47.940820 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:49:47.940892 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":84}
I20260810 04:49:47.940999 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.937084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:34595 (local address 127.14.237.2:43047)
0810 04:49:47.937329 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:47.937340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:47.937415 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:47.937558 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:47.937559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:47.937560 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:47.937635 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.937636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.938558 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.938559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.939887 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:47.939891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.940670 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.940671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:47.940755 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.940762 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 04:49:47.940820 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:49:47.940892 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":88,"thread_start_us":154,"threads_started":1}
I20260810 04:49:47.941495 14013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3b4cb4f1f14fb7a1ebae22016c1900" candidate_uuid: "b944db326faa49e2a7e1d4f07c369da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866cf46374b44bceacb9a5411f911e30" is_pre_election: true
I20260810 04:49:47.941699 14013 raft_consensus.cc:2454] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b944db326faa49e2a7e1d4f07c369da3 in term 0.
I20260810 04:49:47.941759 14450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3b4cb4f1f14fb7a1ebae22016c1900" candidate_uuid: "b944db326faa49e2a7e1d4f07c369da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67420769bcdd43bea5efa45f1fac4405" is_pre_election: true
I20260810 04:49:47.941929 14450 raft_consensus.cc:2454] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b944db326faa49e2a7e1d4f07c369da3 in term 0.
I20260810 04:49:47.941957 14121 leader_election.cc:302] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 866cf46374b44bceacb9a5411f911e30, b944db326faa49e2a7e1d4f07c369da3; no voters:
I20260810 04:49:47.942071 14572 raft_consensus.cc:2781] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:49:47.942119 14572 raft_consensus.cc:482] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:49:47.942147 14572 raft_consensus.cc:3037] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:47.942914 14572 raft_consensus.cc:504] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.943066 14572 leader_election.cc:288] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [CANDIDATE]: Term 1 election: Requested vote from peers 866cf46374b44bceacb9a5411f911e30 (127.14.237.1:42003), 67420769bcdd43bea5efa45f1fac4405 (127.14.237.3:34595)
I20260810 04:49:47.943228 14450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3b4cb4f1f14fb7a1ebae22016c1900" candidate_uuid: "b944db326faa49e2a7e1d4f07c369da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67420769bcdd43bea5efa45f1fac4405"
I20260810 04:49:47.943220 14013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3b4cb4f1f14fb7a1ebae22016c1900" candidate_uuid: "b944db326faa49e2a7e1d4f07c369da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866cf46374b44bceacb9a5411f911e30"
I20260810 04:49:47.943305 14450 raft_consensus.cc:3037] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:47.943305 14013 raft_consensus.cc:3037] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:47.944200 14450 raft_consensus.cc:2454] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b944db326faa49e2a7e1d4f07c369da3 in term 1.
I20260810 04:49:47.944200 14013 raft_consensus.cc:2454] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b944db326faa49e2a7e1d4f07c369da3 in term 1.
I20260810 04:49:47.944382 14121 leader_election.cc:302] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 866cf46374b44bceacb9a5411f911e30, b944db326faa49e2a7e1d4f07c369da3; no voters:
I20260810 04:49:47.944500 14572 raft_consensus.cc:2781] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:47.944653 14572 raft_consensus.cc:686] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 1 LEADER]: Becoming Leader. State: Replica: b944db326faa49e2a7e1d4f07c369da3, State: Running, Role: LEADER
I20260810 04:49:47.944762 14572 consensus_queue.cc:237] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } }
I20260810 04:49:47.945590 13787 catalog_manager.cc:5495] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b944db326faa49e2a7e1d4f07c369da3 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "b944db326faa49e2a7e1d4f07c369da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b944db326faa49e2a7e1d4f07c369da3" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 41135 } health_report { overall_health: HEALTHY } } }
I20260810 04:49:47.958647 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:47.956383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.14.237.2:41135)
0810 04:49:47.956442 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:47.956444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:47.956452 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:49:47.956531 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:47.956532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:47.956547 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:47.956605 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:49:47.956732 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.957168 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.957921 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:47.958098 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:47.958147 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:47.958287 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:49:47.958369 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:49:47.958438 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:47.958483 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 04:49:47.958492 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:49:47.960398 14450 raft_consensus.cc:1261] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 1 FOLLOWER]: Refusing update from remote peer b944db326faa49e2a7e1d4f07c369da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:47.960421 14013 raft_consensus.cc:1261] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 866cf46374b44bceacb9a5411f911e30 [term 1 FOLLOWER]: Refusing update from remote peer b944db326faa49e2a7e1d4f07c369da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:47.960731 14577 consensus_queue.cc:1035] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "866cf46374b44bceacb9a5411f911e30" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:47.960860 14572 consensus_queue.cc:1035] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67420769bcdd43bea5efa45f1fac4405" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:47.963501 14584 mvcc.cc:205] Tried to move back new op lower bound from 7316837941083095040 to 7316837941022248960. Current Snapshot: MvccSnapshot[applied={T|T < 7316837941083095040}]
I20260810 04:49:47.963624 14588 mvcc.cc:205] Tried to move back new op lower bound from 7316837941083095040 to 7316837941022248960. Current Snapshot: MvccSnapshot[applied={T|T < 7316837941083095040}]
I20260810 04:49:47.964279 14589 mvcc.cc:205] Tried to move back new op lower bound from 7316837941083095040 to 7316837941022248960. Current Snapshot: MvccSnapshot[applied={T|T < 7316837941083095040}]
I20260810 04:49:47.966409 13786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:47.966506 13786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:47.967253 13786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19d304bdcbc84f29b723d759ddaf9360: Sending DeleteTablet for 3 replicas of tablet 2b3b4cb4f1f14fb7a1ebae22016c1900
I20260810 04:49:47.967588 14423 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:47 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:50054
I20260810 04:49:47.967574 13983 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:47 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:41356
I20260810 04:49:47.967824 14603 tablet_replica.cc:330] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: stopping tablet replica
I20260810 04:49:47.967933 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 13891
I20260810 04:49:47.967938 14202 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:47 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:46974
I20260810 04:49:48.002939 14603 raft_consensus.cc:2229] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:48.003176 14603 raft_consensus.cc:2258] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:48.003970 14603 ts_tablet_manager.cc:1905] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:48.005378 14606 tablet_replica.cc:330] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: stopping tablet replica
I20260810 04:49:48.005507 14606 raft_consensus.cc:2229] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:49:48.005646 14606 raft_consensus.cc:2258] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:48.005658 14603 ts_tablet_manager.cc:1918] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:48.005728 14603 log.cc:1196] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/2b3b4cb4f1f14fb7a1ebae22016c1900
I20260810 04:49:48.006022 14603 ts_tablet_manager.cc:1939] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P 67420769bcdd43bea5efa45f1fac4405: Deleting consensus metadata
I20260810 04:49:48.006520 13770 catalog_manager.cc:4841] TS 67420769bcdd43bea5efa45f1fac4405 (127.14.237.3:34595): tablet 2b3b4cb4f1f14fb7a1ebae22016c1900 (table test-table [id=073f17cdec7c4f00b76cb09a265072f0]) successfully deleted
I20260810 04:49:48.006735 14606 ts_tablet_manager.cc:1905] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:48.008251 14606 ts_tablet_manager.cc:1918] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:48.008327 14606 log.cc:1196] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/2b3b4cb4f1f14fb7a1ebae22016c1900
I20260810 04:49:48.008579 14606 ts_tablet_manager.cc:1939] T 2b3b4cb4f1f14fb7a1ebae22016c1900 P b944db326faa49e2a7e1d4f07c369da3: Deleting consensus metadata
I20260810 04:49:48.009018 13766 catalog_manager.cc:4841] TS b944db326faa49e2a7e1d4f07c369da3 (127.14.237.2:41135): tablet 2b3b4cb4f1f14fb7a1ebae22016c1900 (table test-table [id=073f17cdec7c4f00b76cb09a265072f0]) successfully deleted
W20260810 04:49:48.010365 13767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:42003 (error 108)
I20260810 04:49:48.010949 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.010801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42003 (local address 127.0.0.1:41360)
0810 04:49:48.010868 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.010908 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 04:49:48.011059 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 14091
W20260810 04:49:48.011077 13767 catalog_manager.cc:4584] TS 866cf46374b44bceacb9a5411f911e30 (127.14.237.1:42003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b3b4cb4f1f14fb7a1ebae22016c1900: Network error: Client connection negotiation failed: client connection to 127.14.237.1:42003: connect: Connection refused (error 111)
I20260810 04:49:48.020223 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 14316
I20260810 04:49:48.031294 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 13736
2026-08-10T04:49:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2214 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T04:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:48Z Disabled control of system clock
I20260810 04:49:48.067842 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:43741
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:43741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:48.147606 14615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:48.147805 14615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:48.147825 14615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:48.148932 14615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:48.149008 14615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:48.149040 14615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:48.149086 14615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:48.149114 14615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:48.150454 14615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:43741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:43741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:48.150722 14615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:48.150987 14615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:48.155928 14625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:48.155959 14624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:48.155973 14627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:48.156116 14615 server_base.cc:1034] running on GCE node
I20260810 04:49:48.156373 14615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:48.156680 14615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:48.157852 14615 hybrid_clock.cc:648] HybridClock initialized: now 1786337388157822 us; error 38 us; skew 500 ppm
I20260810 04:49:48.159220 14615 webserver.cc:460] Webserver started at http://127.14.237.62:44609/ using document root <none> and password file <none>
I20260810 04:49:48.159471 14615 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:48.159538 14615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:48.159691 14615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:48.160696 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "bc0ce731d55e443185fa8b7625edbb50"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "9a892a3479f0fbc01ae1aa99fc759302"
server_key_iv: "8e1f086bf863f2894198f67f04e97d89"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.161055 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "bc0ce731d55e443185fa8b7625edbb50"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "9a892a3479f0fbc01ae1aa99fc759302"
server_key_iv: "8e1f086bf863f2894198f67f04e97d89"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.162408 14615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:48.163173 14635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.163342 14615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:48.163443 14615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "bc0ce731d55e443185fa8b7625edbb50"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "9a892a3479f0fbc01ae1aa99fc759302"
server_key_iv: "8e1f086bf863f2894198f67f04e97d89"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.163527 14615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:48.171453 14615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:48.171716 14615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:48.171909 14615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:48.175886 14615 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:43741
I20260810 04:49:48.175916 14721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:43741 every 8 connection(s)
I20260810 04:49:48.176230 14615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:48.177299 14723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:48.179550 14723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50: Bootstrap starting.
I20260810 04:49:48.180142 14723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:48.180444 14723 log.cc:824] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:48.181097 14723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50: No bootstrap required, opened a new log
I20260810 04:49:48.182114 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 14615
I20260810 04:49:48.182238 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:48.182416 15284 external_mini_cluster.cc:1428] Setting key b0a3001e53dad1ea30cb80b3d65fb928
I20260810 04:49:48.182928 14723 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0ce731d55e443185fa8b7625edbb50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43741 } }
I20260810 04:49:48.183118 14723 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:48.183161 14723 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0ce731d55e443185fa8b7625edbb50, State: Initialized, Role: FOLLOWER
I20260810 04:49:48.183332 14723 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0ce731d55e443185fa8b7625edbb50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43741 } }
I20260810 04:49:48.183440 14723 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:48.183486 14723 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:48.183535 14723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:48.184175 14723 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0ce731d55e443185fa8b7625edbb50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43741 } }
I20260810 04:49:48.184265 14723 leader_election.cc:302] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc0ce731d55e443185fa8b7625edbb50; no voters:
I20260810 04:49:48.184454 14723 leader_election.cc:288] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:48.184487 14729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:48.184607 14729 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [term 1 LEADER]: Becoming Leader. State: Replica: bc0ce731d55e443185fa8b7625edbb50, State: Running, Role: LEADER
I20260810 04:49:48.184716 14723 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:48.184688 14729 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0ce731d55e443185fa8b7625edbb50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43741 } }
I20260810 04:49:48.185128 14729 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc0ce731d55e443185fa8b7625edbb50. Latest consensus state: current_term: 1 leader_uuid: "bc0ce731d55e443185fa8b7625edbb50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0ce731d55e443185fa8b7625edbb50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43741 } } }
I20260810 04:49:48.185240 14729 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:48.185137 14730 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc0ce731d55e443185fa8b7625edbb50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0ce731d55e443185fa8b7625edbb50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43741 } } }
I20260810 04:49:48.185393 14730 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50 [sys.catalog]: This master's current role is: LEADER
W20260810 04:49:48.186790 14755 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:49:48.186846 14755 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:49:48.186895 14735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:48.187029 14735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:48.188483 14735 catalog_manager.cc:1347] Generated new cluster ID: 543448e3900d4e4883ddeebecb87bcb1
I20260810 04:49:48.188534 14735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:48.188815 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.182967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.14.237.62:43741)
0810 04:49:48.183249 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.183256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.183303 (+ 47us) server_negotiation.cc:407] Connection header received
0810 04:49:48.183406 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.183411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.183480 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.183619 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:48.183816 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.185524 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.186642 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.187179 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.187309 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.188153 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:48.188157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:48.188157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:48.188330 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:48.188361 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:48.188371 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:48.188451 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.188526 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.188607 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260810 04:49:48.200695 14735 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:48.201324 14735 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:48.209415 14735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50: Generated new TSK 0
I20260810 04:49:48.209578 14735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:49:48.217808 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43741
--builtin_ntp_servers=127.14.237.20:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:48.300567 14763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:48.300798 14763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:48.300834 14763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:48.302040 14763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:48.302100 14763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:48.302160 14763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:49:48.303575 14763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:48.303846 14763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:48.304133 14763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:48.309165 14771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:48.309192 14773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:48.309378 14763 server_base.cc:1034] running on GCE node
W20260810 04:49:48.309186 14775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:48.309698 14763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:48.310001 14763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:48.311188 14763 hybrid_clock.cc:648] HybridClock initialized: now 1786337388311164 us; error 39 us; skew 500 ppm
I20260810 04:49:48.312628 14763 webserver.cc:460] Webserver started at http://127.14.237.1:44071/ using document root <none> and password file <none>
I20260810 04:49:48.312889 14763 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:48.312953 14763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:48.313103 14763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:48.314262 14763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "f1c7bda657b9419d961c3baacda60da8"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "0c5f85ec8a786908ca6f0fad5bf51159"
server_key_iv: "d117cadaa4654dd214ba50f099914ccc"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.314595 14763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "f1c7bda657b9419d961c3baacda60da8"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "0c5f85ec8a786908ca6f0fad5bf51159"
server_key_iv: "d117cadaa4654dd214ba50f099914ccc"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.315848 14763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:49:48.316534 14782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.316700 14763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:48.316790 14763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "f1c7bda657b9419d961c3baacda60da8"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "0c5f85ec8a786908ca6f0fad5bf51159"
server_key_iv: "d117cadaa4654dd214ba50f099914ccc"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.316843 14763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:48.328936 14763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:48.329156 14763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:48.329316 14763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:48.329504 14763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:48.329761 14763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:48.329810 14763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.329845 14763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:48.329857 14763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.335357 14763 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:38063
I20260810 04:49:48.335433 14977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:38063 every 8 connection(s)
I20260810 04:49:48.335742 14763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:49:48.340636 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.336440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:40583 (local address 127.14.237.62:43741)
0810 04:49:48.336506 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.336507 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.336740 (+ 233us) server_negotiation.cc:407] Connection header received
0810 04:49:48.337312 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.337314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.337327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.337380 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:48.338001 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.338465 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.339418 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.339597 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.339646 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.340402 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:48.340404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:48.340404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:48.340408 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:48.340422 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:48.340426 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:48.340468 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.340554 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.340579 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:49:48.340890 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.336410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43741 (local address 127.14.237.1:40583)
0810 04:49:48.336625 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.336656 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:48.337152 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:48.337435 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:48.337438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:48.337446 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:48.337931 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.337935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.338554 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.338565 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.339343 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.339344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.339672 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.339673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.339777 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.340269 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:48.340269 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:48.340298 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:48.340319 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:48.340484 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:48.340489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:48.340491 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:49:48.340543 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:49:48.340628 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260810 04:49:48.341423 14979 heartbeater.cc:347] Connected to a master server at 127.14.237.62:43741
I20260810 04:49:48.341528 14979 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:48.341710 14979 heartbeater.cc:511] Master 127.14.237.62:43741 requested a full tablet report, sending...
I20260810 04:49:48.342288 14657 ts_manager.cc:194] Registered new tserver with Master: f1c7bda657b9419d961c3baacda60da8 (127.14.237.1:38063)
I20260810 04:49:48.342902 14657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.1:40583
I20260810 04:49:48.343490 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 14763
I20260810 04:49:48.343570 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:49:48.343784 15284 external_mini_cluster.cc:1428] Setting key 2675afc6a0524322e045258771df3b73
I20260810 04:49:48.344740 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43741
--builtin_ntp_servers=127.14.237.20:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:48.427037 14984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:48.427234 14984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:48.427253 14984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:48.428265 14984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:48.428295 14984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:48.428319 14984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:49:48.429625 14984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:48.429804 14984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:48.430035 14984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:48.434852 14989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:48.434861 14995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:48.434894 14991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:48.435101 14984 server_base.cc:1034] running on GCE node
I20260810 04:49:48.435396 14984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:48.435581 14984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:48.436743 14984 hybrid_clock.cc:648] HybridClock initialized: now 1786337388436729 us; error 26 us; skew 500 ppm
I20260810 04:49:48.437933 14984 webserver.cc:460] Webserver started at http://127.14.237.2:40711/ using document root <none> and password file <none>
I20260810 04:49:48.438143 14984 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:48.438186 14984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:48.438277 14984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:48.439283 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "dcbe6107f4fe30c1c72ce1ca17fb15d7"
server_key_iv: "16b22d4b4694f2569ffed3adf34eb9ab"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.439599 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "dcbe6107f4fe30c1c72ce1ca17fb15d7"
server_key_iv: "16b22d4b4694f2569ffed3adf34eb9ab"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.440860 14984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:49:48.441695 15003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.441876 14984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.441963 14984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "dcbe6107f4fe30c1c72ce1ca17fb15d7"
server_key_iv: "16b22d4b4694f2569ffed3adf34eb9ab"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.442015 14984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:48.453464 14984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:48.453692 14984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:48.453783 14984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:48.453958 14984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:48.454311 14984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:48.454344 14984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.454393 14984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:48.454444 14984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.460253 14984 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:35167
I20260810 04:49:48.460310 15185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:35167 every 8 connection(s)
I20260810 04:49:48.460592 14984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:49:48.465628 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.461358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:56081 (local address 127.14.237.62:43741)
0810 04:49:48.461445 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.461447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.461610 (+ 163us) server_negotiation.cc:407] Connection header received
0810 04:49:48.462105 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.462106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.462118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.462180 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:48.462853 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.463353 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.464315 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.464516 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.464590 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.465347 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:48.465348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:48.465349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:48.465354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:48.465369 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:48.465373 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:48.465425 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.465526 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.465552 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 04:49:48.465816 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.461275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43741 (local address 127.14.237.2:56081)
0810 04:49:48.461499 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.461529 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:48.461997 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:48.462257 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:48.462260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:48.462268 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:48.462755 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.462759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.463453 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.463454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.464203 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.464205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.464600 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.464600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.464731 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.465165 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:48.465165 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:48.465189 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:48.465255 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:48.465431 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:48.465436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:48.465439 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:48.465515 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:49:48.465611 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260810 04:49:48.466292 15186 heartbeater.cc:347] Connected to a master server at 127.14.237.62:43741
I20260810 04:49:48.466418 15186 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:48.466621 15186 heartbeater.cc:511] Master 127.14.237.62:43741 requested a full tablet report, sending...
I20260810 04:49:48.466998 14657 ts_manager.cc:194] Registered new tserver with Master: 01b325a2d8ff4e4fbf9c493cf2852fd1 (127.14.237.2:35167)
I20260810 04:49:48.467423 14657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.2:56081
I20260810 04:49:48.470207 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 14984
I20260810 04:49:48.470300 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:49:48.470453 15284 external_mini_cluster.cc:1428] Setting key f6944b2dded41aebed06cbe03dd13ffd
I20260810 04:49:48.471215 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43741
--builtin_ntp_servers=127.14.237.20:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:48.550869 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:48.551064 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:48.551102 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:48.552295 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:48.552338 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:48.552362 15192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:49:48.553781 15192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:48.553958 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:48.554220 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:48.559044 15207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:48.559020 15206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:48.559092 15210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:48.559137 15192 server_base.cc:1034] running on GCE node
I20260810 04:49:48.559422 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:48.559661 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:48.560807 15192 hybrid_clock.cc:648] HybridClock initialized: now 1786337388560791 us; error 28 us; skew 500 ppm
I20260810 04:49:48.562129 15192 webserver.cc:460] Webserver started at http://127.14.237.3:45885/ using document root <none> and password file <none>
I20260810 04:49:48.562377 15192 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:48.562450 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:48.562615 15192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:48.563612 15192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "9e8b8b861bdb4b23903ec616e512cdcc"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "58d055cb1e2dd16452ab370b50ac0ab3"
server_key_iv: "20c8e24c947464b4025f249910f72ccc"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.563995 15192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "9e8b8b861bdb4b23903ec616e512cdcc"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "58d055cb1e2dd16452ab370b50ac0ab3"
server_key_iv: "20c8e24c947464b4025f249910f72ccc"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.565316 15192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:49:48.566035 15221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.566188 15192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:48.566286 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "9e8b8b861bdb4b23903ec616e512cdcc"
format_stamp: "Formatted at 2026-08-10 04:49:48 on dist-test-slave-8hhm"
server_key: "58d055cb1e2dd16452ab370b50ac0ab3"
server_key_iv: "20c8e24c947464b4025f249910f72ccc"
server_key_version: "encryptionkey@0"
I20260810 04:49:48.566373 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:48.579070 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:48.579366 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:48.579511 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:48.579730 15192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:48.580029 15192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:48.580082 15192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.580123 15192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:48.580152 15192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:48.586104 15192 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:38599
I20260810 04:49:48.586155 15416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:38599 every 8 connection(s)
I20260810 04:49:48.586474 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:49:48.586671 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 15192
I20260810 04:49:48.586786 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:49:48.587011 15284 external_mini_cluster.cc:1428] Setting key 72fa7fe13407fb4e78811d217a862099
I20260810 04:49:48.591739 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.587278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:44655 (local address 127.14.237.62:43741)
0810 04:49:48.587347 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.587349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.587513 (+ 164us) server_negotiation.cc:407] Connection header received
0810 04:49:48.588123 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.588124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.588136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.588175 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:48.588801 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.589379 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.590347 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.590572 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.590664 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.591491 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:48.591492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:48.591493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:48.591498 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:48.591512 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:48.591516 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:48.591564 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.591635 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.591661 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:49:48.591960 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.587222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:43741 (local address 127.14.237.3:44655)
0810 04:49:48.587412 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.587454 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:48.588026 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:48.588248 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:48.588252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:48.588260 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:48.588711 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.588715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.589469 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.589470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.590254 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.590256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.590676 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.590677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.590818 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.591354 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:48.591355 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:48.591380 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:48.591401 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:48.591564 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:48.591569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:48.591572 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:48.591636 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:49:48.591729 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260810 04:49:48.592458 15422 heartbeater.cc:347] Connected to a master server at 127.14.237.62:43741
I20260810 04:49:48.592535 15422 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:48.592731 15422 heartbeater.cc:511] Master 127.14.237.62:43741 requested a full tablet report, sending...
I20260810 04:49:48.593086 14657 ts_manager.cc:194] Registered new tserver with Master: 9e8b8b861bdb4b23903ec616e512cdcc (127.14.237.3:38599)
I20260810 04:49:48.593468 14657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.3:44655
I20260810 04:49:48.598803 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:49:48.603106 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.600188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.14.237.62:43741)
0810 04:49:48.600278 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.600280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.600396 (+ 116us) server_negotiation.cc:407] Connection header received
0810 04:49:48.600462 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.600463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.600475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.600524 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:48.600687 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.601140 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.601915 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.602122 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.602197 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.602813 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:48.602814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:48.602815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:48.602819 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:48.602832 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:48.602835 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:48.602950 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.603021 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.603032 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:49:48.604682 14657 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:49:48.606134 14657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:49:48.612361 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.608270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:38063 (local address 127.0.0.1:40894)
0810 04:49:48.608704 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.608715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:48.608779 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:48.608953 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:48.608954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:48.608955 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:48.608985 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.608986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.610315 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.610316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.611416 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.611418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.612064 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.612065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.612144 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.612159 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 04:49:48.612235 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:49:48.612290 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260810 04:49:48.612409 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.608517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.14.237.1:38063)
0810 04:49:48.608726 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.608730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.608764 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:49:48.608844 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.608845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.608860 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.609056 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:48.609149 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.610227 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.611499 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.611966 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.612060 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.612160 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.612281 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.612327 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260810 04:49:48.613281 14865 tablet_service.cc:1467] Processing CreateTablet for tablet fa75ab2e68014dfcae94176092bbcd2f (DEFAULT_TABLE table=test-table [id=41b70eea43084a7b8739b6c3653a0f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:48.613469 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.608268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:38599 (local address 127.0.0.1:52020)
0810 04:49:48.608376 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.608412 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:48.608471 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:48.608905 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:48.608908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:48.608910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:48.608976 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.608978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.611522 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.611523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.612184 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.612186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.613208 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.613209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.613299 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.613302 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:48.613362 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:49:48.613401 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":88,"threads_started":1}
I20260810 04:49:48.613483 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.608533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.14.237.3:38599)
0810 04:49:48.608726 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.608730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.608741 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:49:48.608776 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.608777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.608792 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.609052 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:48.609115 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.610226 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.612303 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.613113 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.613205 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.613303 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.613379 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.613414 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260810 04:49:48.613663 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.608270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:35167 (local address 127.0.0.1:56316)
0810 04:49:48.608868 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.608877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:48.608915 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:48.609182 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:48.609183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:48.609185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:48.609251 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.609253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.610118 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.610119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.611415 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.611418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.613427 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.613429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.613492 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.613496 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:48.613546 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:49:48.613585 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":7,"thread_start_us":59,"threads_started":1}
I20260810 04:49:48.613684 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa75ab2e68014dfcae94176092bbcd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:48.613942 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.608734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.14.237.2:35167)
0810 04:49:48.608935 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.608939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.608949 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:49:48.609004 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.609006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.609020 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.609111 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:48.609334 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.610023 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.612201 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.613162 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.613291 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.613783 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.613812 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.613854 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260810 04:49:48.614216 15314 tablet_service.cc:1467] Processing CreateTablet for tablet fa75ab2e68014dfcae94176092bbcd2f (DEFAULT_TABLE table=test-table [id=41b70eea43084a7b8739b6c3653a0f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:48.614490 15314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa75ab2e68014dfcae94176092bbcd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:48.614629 15082 tablet_service.cc:1467] Processing CreateTablet for tablet fa75ab2e68014dfcae94176092bbcd2f (DEFAULT_TABLE table=test-table [id=41b70eea43084a7b8739b6c3653a0f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:48.614938 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa75ab2e68014dfcae94176092bbcd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:48.616808 15446 tablet_bootstrap.cc:492] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: Bootstrap starting.
I20260810 04:49:48.617580 15446 tablet_bootstrap.cc:654] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:48.617863 15448 tablet_bootstrap.cc:492] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8: Bootstrap starting.
I20260810 04:49:48.617920 15446 log.cc:824] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:48.618378 15450 tablet_bootstrap.cc:492] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Bootstrap starting.
I20260810 04:49:48.618649 15448 tablet_bootstrap.cc:654] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:48.618840 15446 tablet_bootstrap.cc:492] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: No bootstrap required, opened a new log
I20260810 04:49:48.618930 15446 ts_tablet_manager.cc:1397] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:49:48.619001 15450 tablet_bootstrap.cc:654] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:48.618997 15448 log.cc:824] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:48.619328 15450 log.cc:824] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:48.619861 15448 tablet_bootstrap.cc:492] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8: No bootstrap required, opened a new log
I20260810 04:49:48.619972 15448 ts_tablet_manager.cc:1397] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:48.620077 15450 tablet_bootstrap.cc:492] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: No bootstrap required, opened a new log
I20260810 04:49:48.620157 15450 ts_tablet_manager.cc:1397] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:49:48.621160 15446 raft_consensus.cc:348] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.621352 15446 raft_consensus.cc:374] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:48.621385 15446 raft_consensus.cc:729] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8b8b861bdb4b23903ec616e512cdcc, State: Initialized, Role: FOLLOWER
I20260810 04:49:48.621487 15446 consensus_queue.cc:260] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.621740 15446 ts_tablet_manager.cc:1428] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:49:48.621763 15422 heartbeater.cc:503] Master 127.14.237.62:43741 was elected leader, sending a full tablet report...
I20260810 04:49:48.622644 15450 raft_consensus.cc:348] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.622808 15450 raft_consensus.cc:374] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:48.622851 15450 raft_consensus.cc:729] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b325a2d8ff4e4fbf9c493cf2852fd1, State: Initialized, Role: FOLLOWER
I20260810 04:49:48.622969 15450 consensus_queue.cc:260] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.623106 15448 raft_consensus.cc:348] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.623232 15450 ts_tablet_manager.cc:1428] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:49:48.623255 15448 raft_consensus.cc:374] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:48.623291 15448 raft_consensus.cc:729] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c7bda657b9419d961c3baacda60da8, State: Initialized, Role: FOLLOWER
I20260810 04:49:48.623286 15186 heartbeater.cc:503] Master 127.14.237.62:43741 was elected leader, sending a full tablet report...
I20260810 04:49:48.623387 15448 consensus_queue.cc:260] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.623641 15448 ts_tablet_manager.cc:1428] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 04:49:48.623704 14979 heartbeater.cc:503] Master 127.14.237.62:43741 was elected leader, sending a full tablet report...
W20260810 04:49:48.711124 15187 tablet.cc:2369] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:49:48.756765 15455 raft_consensus.cc:482] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:48.756873 15455 raft_consensus.cc:504] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.757207 15455 leader_election.cc:288] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1c7bda657b9419d961c3baacda60da8 (127.14.237.1:38063), 9e8b8b861bdb4b23903ec616e512cdcc (127.14.237.3:38599)
I20260810 04:49:48.761003 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.757450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:47017 (local address 127.14.237.1:38063)
0810 04:49:48.757494 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.757496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.757504 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:49:48.757582 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.757583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.757594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.757662 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:48.757866 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.758656 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.760070 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.760623 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.760694 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.760700 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.760877 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.760916 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:49:48.761003 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.757488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:52283 (local address 127.14.237.3:38599)
0810 04:49:48.757532 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.757534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.757648 (+ 114us) server_negotiation.cc:407] Connection header received
0810 04:49:48.757720 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.757721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.757729 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.757773 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:48.757944 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.758629 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.760063 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.760619 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.760693 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.760700 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.760877 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.760916 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:49:48.761003 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.757385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:38063 (local address 127.14.237.2:47017)
0810 04:49:48.757449 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.757460 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:48.757500 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:48.757678 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:48.757679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:48.757681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:48.757753 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.757755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.758723 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.758724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.759947 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.759988 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.760702 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.760703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.760793 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.760797 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:48.760855 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:49:48.760897 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":72}
I20260810 04:49:48.761011 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.757398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:38599 (local address 127.14.237.2:52283)
0810 04:49:48.757575 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.757583 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:48.757658 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:48.757795 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:48.757795 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:48.757797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:48.757873 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.757875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.758728 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.758728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.759951 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:48.759953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.760704 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.760705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:48.760800 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.760803 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:48.760855 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:49:48.760893 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":60,"thread_start_us":99,"threads_started":1}
I20260810 04:49:48.761390 14899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa75ab2e68014dfcae94176092bbcd2f" candidate_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1c7bda657b9419d961c3baacda60da8" is_pre_election: true
I20260810 04:49:48.761539 14899 raft_consensus.cc:2454] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b325a2d8ff4e4fbf9c493cf2852fd1 in term 0.
I20260810 04:49:48.761716 15346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa75ab2e68014dfcae94176092bbcd2f" candidate_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" is_pre_election: true
I20260810 04:49:48.761765 15007 leader_election.cc:302] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b325a2d8ff4e4fbf9c493cf2852fd1, f1c7bda657b9419d961c3baacda60da8; no voters:
I20260810 04:49:48.761875 15346 raft_consensus.cc:2454] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b325a2d8ff4e4fbf9c493cf2852fd1 in term 0.
I20260810 04:49:48.761880 15455 raft_consensus.cc:2781] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:49:48.761926 15455 raft_consensus.cc:482] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:49:48.761960 15455 raft_consensus.cc:3037] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:48.762758 15455 raft_consensus.cc:504] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.762897 15455 leader_election.cc:288] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [CANDIDATE]: Term 1 election: Requested vote from peers f1c7bda657b9419d961c3baacda60da8 (127.14.237.1:38063), 9e8b8b861bdb4b23903ec616e512cdcc (127.14.237.3:38599)
I20260810 04:49:48.763057 15346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa75ab2e68014dfcae94176092bbcd2f" candidate_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8b8b861bdb4b23903ec616e512cdcc"
I20260810 04:49:48.763134 15346 raft_consensus.cc:3037] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:48.763195 14899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa75ab2e68014dfcae94176092bbcd2f" candidate_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1c7bda657b9419d961c3baacda60da8"
I20260810 04:49:48.763278 14899 raft_consensus.cc:3037] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:48.763921 15346 raft_consensus.cc:2454] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b325a2d8ff4e4fbf9c493cf2852fd1 in term 1.
I20260810 04:49:48.764103 14899 raft_consensus.cc:2454] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b325a2d8ff4e4fbf9c493cf2852fd1 in term 1.
I20260810 04:49:48.764137 15006 leader_election.cc:302] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b325a2d8ff4e4fbf9c493cf2852fd1, 9e8b8b861bdb4b23903ec616e512cdcc; no voters:
I20260810 04:49:48.764245 15455 raft_consensus.cc:2781] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:48.764369 15455 raft_consensus.cc:686] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 01b325a2d8ff4e4fbf9c493cf2852fd1, State: Running, Role: LEADER
I20260810 04:49:48.764461 15455 consensus_queue.cc:237] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } }
I20260810 04:49:48.765282 14656 catalog_manager.cc:5495] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01b325a2d8ff4e4fbf9c493cf2852fd1 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01b325a2d8ff4e4fbf9c493cf2852fd1" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 35167 } health_report { overall_health: HEALTHY } } }
I20260810 04:49:48.796391 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.794305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.14.237.2:35167)
0810 04:49:48.794344 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:48.794346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:48.794353 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:49:48.794411 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:48.794412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:48.794429 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:48.794486 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:49:48.794604 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.795028 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.795686 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:48.795860 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:48.795913 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:48.796035 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:49:48.796120 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:49:48.796191 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:48.796248 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 04:49:48.796255 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:49:48.798162 15346 raft_consensus.cc:1261] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 1 FOLLOWER]: Refusing update from remote peer 01b325a2d8ff4e4fbf9c493cf2852fd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:48.798166 14899 raft_consensus.cc:1261] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [term 1 FOLLOWER]: Refusing update from remote peer 01b325a2d8ff4e4fbf9c493cf2852fd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:48.798496 15455 consensus_queue.cc:1035] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8b8b861bdb4b23903ec616e512cdcc" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:48.798668 15461 consensus_queue.cc:1035] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1c7bda657b9419d961c3baacda60da8" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 38063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:48.801422 15467 mvcc.cc:205] Tried to move back new op lower bound from 7316837944514744320 to 7316837944379736064. Current Snapshot: MvccSnapshot[applied={T|T < 7316837944514744320}]
I20260810 04:49:48.801544 15465 mvcc.cc:205] Tried to move back new op lower bound from 7316837944514744320 to 7316837944379736064. Current Snapshot: MvccSnapshot[applied={T|T < 7316837944514744320}]
I20260810 04:49:48.803128 14656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:48.803246 14656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:48.803994 14656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc0ce731d55e443185fa8b7625edbb50: Sending DeleteTablet for 3 replicas of tablet fa75ab2e68014dfcae94176092bbcd2f
I20260810 04:49:48.804329 15314 tablet_service.cc:1514] Processing DeleteTablet for tablet fa75ab2e68014dfcae94176092bbcd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:48 UTC) from {username='slave'} at 127.0.0.1:52020
I20260810 04:49:48.804327 14865 tablet_service.cc:1514] Processing DeleteTablet for tablet fa75ab2e68014dfcae94176092bbcd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:48 UTC) from {username='slave'} at 127.0.0.1:40894
I20260810 04:49:48.804349 15082 tablet_service.cc:1514] Processing DeleteTablet for tablet fa75ab2e68014dfcae94176092bbcd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:48 UTC) from {username='slave'} at 127.0.0.1:56316
I20260810 04:49:48.804549 15483 tablet_replica.cc:330] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8: stopping tablet replica
I20260810 04:49:48.804601 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 14763
I20260810 04:49:48.804651 15483 raft_consensus.cc:2229] T fa75ab2e68014dfcae94176092bbcd2f P f1c7bda657b9419d961c3baacda60da8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:48.812353 15484 tablet_replica.cc:330] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: stopping tablet replica
I20260810 04:49:48.812455 15484 raft_consensus.cc:2229] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:49:48.812579 15484 raft_consensus.cc:2258] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:48.812736 15480 tablet_replica.cc:330] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: stopping tablet replica
I20260810 04:49:48.812824 15480 raft_consensus.cc:2229] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:48.812911 15480 raft_consensus.cc:2258] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:48.813148 15484 ts_tablet_manager.cc:1905] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:48.815380 15484 ts_tablet_manager.cc:1918] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:48.815465 15484 log.cc:1196] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/fa75ab2e68014dfcae94176092bbcd2f
I20260810 04:49:48.815737 15484 ts_tablet_manager.cc:1939] T fa75ab2e68014dfcae94176092bbcd2f P 01b325a2d8ff4e4fbf9c493cf2852fd1: Deleting consensus metadata
I20260810 04:49:48.815982 15480 ts_tablet_manager.cc:1905] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:48.816126 14636 catalog_manager.cc:4841] TS 01b325a2d8ff4e4fbf9c493cf2852fd1 (127.14.237.2:35167): tablet fa75ab2e68014dfcae94176092bbcd2f (table test-table [id=41b70eea43084a7b8739b6c3653a0f27]) successfully deleted
I20260810 04:49:48.817844 15480 ts_tablet_manager.cc:1918] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:48.817924 15480 log.cc:1196] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/fa75ab2e68014dfcae94176092bbcd2f
I20260810 04:49:48.818169 15480 ts_tablet_manager.cc:1939] T fa75ab2e68014dfcae94176092bbcd2f P 9e8b8b861bdb4b23903ec616e512cdcc: Deleting consensus metadata
I20260810 04:49:48.818629 14637 catalog_manager.cc:4841] TS 9e8b8b861bdb4b23903ec616e512cdcc (127.14.237.3:38599): tablet fa75ab2e68014dfcae94176092bbcd2f (table test-table [id=41b70eea43084a7b8739b6c3653a0f27]) successfully deleted
W20260810 04:49:48.819458 14638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:38063 (error 108)
I20260810 04:49:48.819816 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 14984
I20260810 04:49:48.819993 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:48.819861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:38063 (local address 127.0.0.1:40906)
0810 04:49:48.819914 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:48.819956 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:38063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:49:48.828838 14638 catalog_manager.cc:4584] TS f1c7bda657b9419d961c3baacda60da8 (127.14.237.1:38063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa75ab2e68014dfcae94176092bbcd2f: Network error: Client connection negotiation failed: client connection to 127.14.237.1:38063: connect: Connection refused (error 111)
I20260810 04:49:48.833014 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 15192
I20260810 04:49:48.840739 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 14615
2026-08-10T04:49:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (807 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:48 dist-test-slave-8hhm krb5kdc[15490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:48 dist-test-slave-8hhm krb5kdc[15490](info): set up 2 sockets
Aug 10 04:49:48 dist-test-slave-8hhm krb5kdc[15490](info): commencing operation
krb5kdc: starting...
W20260810 04:49:50.084982 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:49:50.201993 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:44053
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:50.281651 15507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:50.281860 15507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:50.281893 15507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:50.283068 15507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:50.283145 15507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:50.283169 15507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:50.283197 15507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:50.283226 15507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:50.284725 15507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:44053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:50.284991 15507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:50.285234 15507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:50.290012 15517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:50.289987 15520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:50.289981 15516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:50.290288 15507 server_base.cc:1034] running on GCE node
I20260810 04:49:50.290530 15507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:50.290802 15507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:50.291980 15507 hybrid_clock.cc:648] HybridClock initialized: now 1786337390291970 us; error 40 us; skew 500 ppm
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:50.294723 15507 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:49:50.295212 15507 webserver.cc:460] Webserver started at http://127.14.237.62:42047/ using document root <none> and password file <none>
I20260810 04:49:50.295460 15507 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:50.295531 15507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:50.295682 15507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:50.296686 15507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "999abe69335849ee826b68e4d125e74b"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "f66032a9d03048cc1036e3e8c48a404d"
server_key_iv: "b422334497dfa9dbc333b09190bb11fc"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.297060 15507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "999abe69335849ee826b68e4d125e74b"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "f66032a9d03048cc1036e3e8c48a404d"
server_key_iv: "b422334497dfa9dbc333b09190bb11fc"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.298405 15507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:50.299059 15534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.299234 15507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.299329 15507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "999abe69335849ee826b68e4d125e74b"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "f66032a9d03048cc1036e3e8c48a404d"
server_key_iv: "b422334497dfa9dbc333b09190bb11fc"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.299404 15507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:50.324321 15507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:50.326761 15507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:50.326923 15507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:50.331116 15507 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:44053
I20260810 04:49:50.331148 15616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:44053 every 8 connection(s)
I20260810 04:49:50.331475 15507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:50.332324 15617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:50.334540 15617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b: Bootstrap starting.
I20260810 04:49:50.335036 15617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:50.335274 15617 log.cc:824] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:50.335898 15617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b: No bootstrap required, opened a new log
I20260810 04:49:50.336830 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 15507
I20260810 04:49:50.336957 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:50.337179 15284 external_mini_cluster.cc:1428] Setting key dc4a1883fa1a62e63a1cc9c2eea06a67
I20260810 04:49:50.337733 15617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999abe69335849ee826b68e4d125e74b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44053 } }
I20260810 04:49:50.337926 15617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:50.337966 15617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999abe69335849ee826b68e4d125e74b, State: Initialized, Role: FOLLOWER
I20260810 04:49:50.338088 15617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999abe69335849ee826b68e4d125e74b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44053 } }
I20260810 04:49:50.338161 15617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:50.338202 15617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:50.338267 15617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:50.339356 15617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999abe69335849ee826b68e4d125e74b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44053 } }
I20260810 04:49:50.339448 15617 leader_election.cc:302] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999abe69335849ee826b68e4d125e74b; no voters:
I20260810 04:49:50.339629 15617 leader_election.cc:288] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:50.339681 15625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:50.339818 15625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [term 1 LEADER]: Becoming Leader. State: Replica: 999abe69335849ee826b68e4d125e74b, State: Running, Role: LEADER
I20260810 04:49:50.339877 15617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:50.339900 15625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999abe69335849ee826b68e4d125e74b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44053 } }
I20260810 04:49:50.340265 15625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999abe69335849ee826b68e4d125e74b. Latest consensus state: current_term: 1 leader_uuid: "999abe69335849ee826b68e4d125e74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999abe69335849ee826b68e4d125e74b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44053 } } }
I20260810 04:49:50.340325 15625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:50.340502 15630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:50.340633 15627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "999abe69335849ee826b68e4d125e74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999abe69335849ee826b68e4d125e74b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44053 } } }
I20260810 04:49:50.340788 15627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:50.340988 15630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:50.343392 15630 catalog_manager.cc:1347] Generated new cluster ID: 023e43d0fce546b1a01d14502fd60472
I20260810 04:49:50.343447 15630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:50.348786 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.337713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.14.237.62:44053)
0810 04:49:50.337913 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.337922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.337955 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:49:50.338064 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.338071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.338122 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.338265 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:50.339192 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.340472 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.341612 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.342135 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.342215 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.344920 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:50.344925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:50.344925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:50.344937 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:50.346992 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.347389 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.347392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.347393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.347519 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.347735 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.347735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.347736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.348042 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:50.348217 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.348284 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.348421 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260810 04:49:50.355141 15630 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:50.355628 15630 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:50.362986 15630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b: Generated new TSK 0
I20260810 04:49:50.363180 15630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.237.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
Entry for principal kudu/127.14.237.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab.
I20260810 04:49:50.397487 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44053
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:50.478271 15655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:50.478474 15655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:50.478497 15655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:50.479988 15655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:50.480038 15655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:50.480082 15655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:49:50.481467 15655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.1.keytab
--principal=kudu/127.14.237.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:50.481704 15655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:50.481915 15655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:50.486850 15663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:50.486876 15655 server_base.cc:1034] running on GCE node
W20260810 04:49:50.486934 15662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:50.486845 15666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:50.487258 15655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:50.487491 15655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:50.488631 15655 hybrid_clock.cc:648] HybridClock initialized: now 1786337390488613 us; error 32 us; skew 500 ppm
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:50.491175 15655 init.cc:377] Logged in from keytab as kudu/127.14.237.1@KRBTEST.COM (short username kudu)
I20260810 04:49:50.491577 15655 webserver.cc:460] Webserver started at http://127.14.237.1:33605/ using document root <none> and password file <none>
I20260810 04:49:50.491801 15655 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:50.491870 15655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:50.492022 15655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:50.493019 15655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "997cecab74224ed98db66251b0c84a6a"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "97c5093f69f21cba4fce987fb5f57fd7"
server_key_iv: "0a96e3603b0ba7d3106fe02120009890"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.493407 15655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "997cecab74224ed98db66251b0c84a6a"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "97c5093f69f21cba4fce987fb5f57fd7"
server_key_iv: "0a96e3603b0ba7d3106fe02120009890"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.494695 15655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:50.495393 15679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.495575 15655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:50.495687 15655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "997cecab74224ed98db66251b0c84a6a"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "97c5093f69f21cba4fce987fb5f57fd7"
server_key_iv: "0a96e3603b0ba7d3106fe02120009890"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.495777 15655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:50.501931 15655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:50.503757 15655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:50.503917 15655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:50.504129 15655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:50.504415 15655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:50.504467 15655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.504504 15655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:50.504530 15655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.509893 15655 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:42355
I20260810 04:49:50.509982 15866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:42355 every 8 connection(s)
I20260810 04:49:50.510237 15655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:49:50.512324 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 15655
I20260810 04:49:50.512445 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:49:50.512650 15284 external_mini_cluster.cc:1428] Setting key bdef231543d8369065e4b2559fdf55fd
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.1@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:50.519520 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.510899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:44345 (local address 127.14.237.62:44053)
0810 04:49:50.510971 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.510973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.511117 (+ 144us) server_negotiation.cc:407] Connection header received
0810 04:49:50.511541 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.511542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.511555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.511601 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:50.512498 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.513117 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.514183 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.514420 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.514507 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.516626 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:50.516627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:50.516628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:50.516635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:50.518124 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.518556 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.518557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.518557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.518623 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.518880 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.518881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.518882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.518998 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:50.519067 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.519337 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.519421 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:49:50.519685 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.510864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:44053 (local address 127.14.237.1:44345)
0810 04:49:50.511008 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.511039 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:50.511466 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:50.511659 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:50.511661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:50.512198 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:50.512366 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.512370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.513234 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.513235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.514078 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.514080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.514517 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.514519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.514665 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.515196 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:50.515203 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:50.516520 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:50.518234 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:50.518241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:50.518243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:50.518463 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:50.518691 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:50.518692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:50.518692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:50.518806 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:50.519072 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:50.519079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:50.519209 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 04:49:50.519313 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 04:49:50.519457 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260810 04:49:50.520179 15867 heartbeater.cc:347] Connected to a master server at 127.14.237.62:44053
I20260810 04:49:50.520268 15867 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:50.520448 15867 heartbeater.cc:511] Master 127.14.237.62:44053 requested a full tablet report, sending...
I20260810 04:49:50.521059 15557 ts_manager.cc:194] Registered new tserver with Master: 997cecab74224ed98db66251b0c84a6a (127.14.237.1:42355)
WARNING: no policy specified for kudu/127.14.237.2@KRBTEST.COM; defaulting to no policy
I20260810 04:49:50.521967 15557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.1@KRBTEST.COM'} at 127.14.237.1:44345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
Entry for principal kudu/127.14.237.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab.
I20260810 04:49:50.539494 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44053
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:50.619340 15876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:50.619521 15876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:50.619544 15876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:50.620549 15876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:50.620582 15876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:50.620606 15876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:49:50.622035 15876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.2.keytab
--principal=kudu/127.14.237.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:50.622205 15876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:50.622355 15876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:50.627326 15887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:50.627313 15884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:50.627313 15882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:50.627554 15876 server_base.cc:1034] running on GCE node
I20260810 04:49:50.627735 15876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:50.627903 15876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:50.629034 15876 hybrid_clock.cc:648] HybridClock initialized: now 1786337390629028 us; error 28 us; skew 500 ppm
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:50.631596 15876 init.cc:377] Logged in from keytab as kudu/127.14.237.2@KRBTEST.COM (short username kudu)
I20260810 04:49:50.632035 15876 webserver.cc:460] Webserver started at http://127.14.237.2:40387/ using document root <none> and password file <none>
I20260810 04:49:50.632228 15876 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:50.632269 15876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:50.632357 15876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:50.633339 15876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "f2ac67c09a9246fe9f8e33b10b775398"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "b66ee969aaba7b986fed1f995b6fb0be"
server_key_iv: "d515df4292c5854bfba5b3d3fef51efc"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.633654 15876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "f2ac67c09a9246fe9f8e33b10b775398"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "b66ee969aaba7b986fed1f995b6fb0be"
server_key_iv: "d515df4292c5854bfba5b3d3fef51efc"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.634940 15876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:50.635701 15902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.635890 15876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:50.635962 15876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "f2ac67c09a9246fe9f8e33b10b775398"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "b66ee969aaba7b986fed1f995b6fb0be"
server_key_iv: "d515df4292c5854bfba5b3d3fef51efc"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.636050 15876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:50.649689 15876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:50.651932 15876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:50.652089 15876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:50.652303 15876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:50.652606 15876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:50.652664 15876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.652705 15876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:50.652733 15876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.658529 15876 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:42771
I20260810 04:49:50.658609 16090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:42771 every 8 connection(s)
I20260810 04:49:50.658934 15876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:49:50.664422 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 15876
I20260810 04:49:50.664552 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:49:50.664789 15284 external_mini_cluster.cc:1428] Setting key 9c44c343809051b245c735b371459a94
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.2@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:50.669687 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.659622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:44053 (local address 127.14.237.2:57431)
0810 04:49:50.659802 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.659847 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:50.660350 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:50.660574 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:50.660577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:50.661186 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:50.661459 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.661465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.662096 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.662097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.662952 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.662954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.663309 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.663310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.663422 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.663838 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:50.663846 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:50.665979 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:50.668313 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:50.668320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:50.668322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:50.668559 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:50.668790 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:50.668791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:50.668791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:50.668886 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:50.669186 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:50.669190 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:50.669373 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 04:49:50.669450 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:49:50.669553 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260810 04:49:50.669700 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.659680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:57431 (local address 127.14.237.62:44053)
0810 04:49:50.659731 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.659733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.659947 (+ 214us) server_negotiation.cc:407] Connection header received
0810 04:49:50.660438 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.660439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.660449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.660504 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:50.661547 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.662012 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.663032 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.663231 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.663282 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.666104 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:50.666105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:50.666106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:50.666114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:50.668190 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.668635 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.668636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.668637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.668710 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.668953 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.668954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.668955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.669115 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:50.669193 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.669501 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.669590 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:49:50.670289 16091 heartbeater.cc:347] Connected to a master server at 127.14.237.62:44053
I20260810 04:49:50.670398 16091 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:50.670559 16091 heartbeater.cc:511] Master 127.14.237.62:44053 requested a full tablet report, sending...
I20260810 04:49:50.670925 15557 ts_manager.cc:194] Registered new tserver with Master: f2ac67c09a9246fe9f8e33b10b775398 (127.14.237.2:42771)
I20260810 04:49:50.671452 15557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.2@KRBTEST.COM'} at 127.14.237.2:57431
WARNING: no policy specified for kudu/127.14.237.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
Entry for principal kudu/127.14.237.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab.
I20260810 04:49:50.691994 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44053
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:50.772840 16097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:50.773020 16097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:50.773042 16097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:50.774113 16097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:50.774147 16097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:50.774173 16097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:49:50.775480 16097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.3.keytab
--principal=kudu/127.14.237.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:50.775643 16097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:50.775800 16097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:50.780771 16110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:50.780763 16109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:50.780834 16097 server_base.cc:1034] running on GCE node
W20260810 04:49:50.780771 16112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:50.781183 16097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:50.781489 16097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:50.782691 16097 hybrid_clock.cc:648] HybridClock initialized: now 1786337390782666 us; error 34 us; skew 500 ppm
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:50.785377 16097 init.cc:377] Logged in from keytab as kudu/127.14.237.3@KRBTEST.COM (short username kudu)
I20260810 04:49:50.785844 16097 webserver.cc:460] Webserver started at http://127.14.237.3:44145/ using document root <none> and password file <none>
I20260810 04:49:50.786124 16097 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:50.786190 16097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:50.786324 16097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:50.787375 16097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "638cddcbf0734a628da45c6251d88db3"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "59fd611e2a790f75a8da8fe461b67d28"
server_key_iv: "66dd4b1c35e1b553341e9d32a02987c6"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.787744 16097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "638cddcbf0734a628da45c6251d88db3"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "59fd611e2a790f75a8da8fe461b67d28"
server_key_iv: "66dd4b1c35e1b553341e9d32a02987c6"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.789115 16097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:49:50.789882 16124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.790053 16097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:50.790182 16097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "638cddcbf0734a628da45c6251d88db3"
format_stamp: "Formatted at 2026-08-10 04:49:50 on dist-test-slave-8hhm"
server_key: "59fd611e2a790f75a8da8fe461b67d28"
server_key_iv: "66dd4b1c35e1b553341e9d32a02987c6"
server_key_version: "encryptionkey@0"
I20260810 04:49:50.790268 16097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:50.802286 16097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:50.804132 16097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:50.804296 16097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:50.804512 16097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:50.804813 16097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:50.804862 16097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.804898 16097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:50.804922 16097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:50.810572 16097 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:40141
I20260810 04:49:50.810627 16296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:40141 every 8 connection(s)
I20260810 04:49:50.810914 16097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:49:50 dist-test-slave-8hhm krb5kdc[15490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337390, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.3@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:49:50.816552 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16097
I20260810 04:49:50.816684 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:49:50.816885 15284 external_mini_cluster.cc:1428] Setting key 73d74b340053255f82f0a5ce4b9c5702
I20260810 04:49:50.819844 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.811598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:42863 (local address 127.14.237.62:44053)
0810 04:49:50.811676 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.811678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.811883 (+ 205us) server_negotiation.cc:407] Connection header received
0810 04:49:50.812372 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.812373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.812383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.812435 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:50.813308 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.813782 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.814689 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.814866 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.814919 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.816988 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:50.816990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:50.816990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:50.816997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:50.818526 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.818896 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.818897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.818898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.818958 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.819325 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.819326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.819326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.819435 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:50.819490 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.819720 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.819784 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:49:50.819991 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.811547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:44053 (local address 127.14.237.3:42863)
0810 04:49:50.811736 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.811802 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:50.812278 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:50.812486 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:50.812489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:50.813041 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:50.813240 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.813245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.813875 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.813876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.814608 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.814610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.814941 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.814941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.815031 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.815434 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:49:50.815442 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:50.816891 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:50.818598 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:50.818603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:50.818605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:50.818810 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:50.819138 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:49:50.819140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:49:50.819140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:49:50.819255 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:49:50.819496 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:50.819501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:50.819606 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 04:49:50.819687 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:49:50.819799 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260810 04:49:50.820413 16298 heartbeater.cc:347] Connected to a master server at 127.14.237.62:44053
I20260810 04:49:50.820520 16298 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:50.820703 16298 heartbeater.cc:511] Master 127.14.237.62:44053 requested a full tablet report, sending...
I20260810 04:49:50.821081 15557 ts_manager.cc:194] Registered new tserver with Master: 638cddcbf0734a628da45c6251d88db3 (127.14.237.3:40141)
I20260810 04:49:50.821496 15557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.237.3@KRBTEST.COM'} at 127.14.237.3:42863
I20260810 04:49:50.828541 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:49:50.835331 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.829859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.14.237.62:44053)
0810 04:49:50.829919 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.829921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.830051 (+ 130us) server_negotiation.cc:407] Connection header received
0810 04:49:50.830089 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.830090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.830098 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.830148 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:50.830651 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.831089 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.831822 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.831992 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.832041 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.833208 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:50.833209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:50.833257 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:50.833264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:50.834424 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.834724 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.834725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.834725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.834773 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:50.834928 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:50.834929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:50.834929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:50.835049 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:50.835141 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.835201 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.835254 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:49:50.836838 15557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:49:50.838403 15557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:49:50.844817 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.840822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.14.237.1:42355)
0810 04:49:50.841049 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.841053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.841187 (+ 134us) server_negotiation.cc:407] Connection header received
0810 04:49:50.841296 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.841298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.841313 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.841388 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:50.841619 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.842272 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.843791 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.844387 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.844476 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.844539 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.844705 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.844748 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260810 04:49:50.844817 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.840700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42355 (local address 127.0.0.1:37434)
0810 04:49:50.841087 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.841098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:50.841192 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:50.841470 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:50.841471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:50.841472 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:50.841525 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.841526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.842369 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.842370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.843709 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.843711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.844476 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.844477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.844564 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.844576 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 04:49:50.844684 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 04:49:50.844738 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":31,"thread_start_us":59,"threads_started":1}
I20260810 04:49:50.844952 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.840682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:40141 (local address 127.0.0.1:35572)
0810 04:49:50.840836 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.840875 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:50.840954 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:50.841355 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:50.841359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:50.841361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:50.841422 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.841424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.842611 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.842612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.843696 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.843699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.844624 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.844625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.844686 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.844689 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:50.844742 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:49:50.844786 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":2,"thread_start_us":52,"threads_started":1}
I20260810 04:49:50.844990 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.840888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.14.237.3:40141)
0810 04:49:50.841060 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.841064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.841075 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:49:50.841161 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.841164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.841181 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.841271 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:50.841501 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.842509 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.843801 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.844466 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.844534 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.844584 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.844858 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.844901 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260810 04:49:50.845525 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.840730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:42771 (local address 127.0.0.1:49336)
0810 04:49:50.841127 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.841138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:50.841194 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:50.841665 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:50.841666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:50.841668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:50.841719 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.841721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.842858 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.842859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.843931 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.843934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.844852 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.844853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.845327 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.845332 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:49:50.845423 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:49:50.845463 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":3,"thread_start_us":39,"threads_started":1}
I20260810 04:49:50.845553 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.840896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.14.237.2:42771)
0810 04:49:50.841070 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.841074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.841344 (+ 270us) server_negotiation.cc:407] Connection header received
0810 04:49:50.841439 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.841442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.841458 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.841584 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:50.841792 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.842761 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.844015 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.844712 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.844772 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.844819 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.845442 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.845482 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260810 04:49:50.845548 15755 tablet_service.cc:1467] Processing CreateTablet for tablet 38e86794994d4aa78ad41351ce18e1c6 (DEFAULT_TABLE table=test-table [id=27af13e53db744d8830ee5388cc624fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:50.846026 15755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e86794994d4aa78ad41351ce18e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:50.848628 16322 tablet_bootstrap.cc:492] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a: Bootstrap starting.
I20260810 04:49:50.848985 16203 tablet_service.cc:1467] Processing CreateTablet for tablet 38e86794994d4aa78ad41351ce18e1c6 (DEFAULT_TABLE table=test-table [id=27af13e53db744d8830ee5388cc624fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:50.849457 16322 tablet_bootstrap.cc:654] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:50.849481 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e86794994d4aa78ad41351ce18e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:50.849953 16322 log.cc:824] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:50.849871 15982 tablet_service.cc:1467] Processing CreateTablet for tablet 38e86794994d4aa78ad41351ce18e1c6 (DEFAULT_TABLE table=test-table [id=27af13e53db744d8830ee5388cc624fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:50.850191 15982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e86794994d4aa78ad41351ce18e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:50.850689 16322 tablet_bootstrap.cc:492] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a: No bootstrap required, opened a new log
I20260810 04:49:50.850773 16322 ts_tablet_manager.cc:1397] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:50.852131 16322 raft_consensus.cc:348] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.852344 16322 raft_consensus.cc:374] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:50.852393 16322 raft_consensus.cc:729] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997cecab74224ed98db66251b0c84a6a, State: Initialized, Role: FOLLOWER
I20260810 04:49:50.852504 16322 consensus_queue.cc:260] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.852818 16322 ts_tablet_manager.cc:1428] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:50.852828 15867 heartbeater.cc:503] Master 127.14.237.62:44053 was elected leader, sending a full tablet report...
I20260810 04:49:50.853060 16325 tablet_bootstrap.cc:492] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: Bootstrap starting.
I20260810 04:49:50.853739 16325 tablet_bootstrap.cc:654] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:50.854005 16326 tablet_bootstrap.cc:492] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: Bootstrap starting.
I20260810 04:49:50.854292 16325 log.cc:824] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:50.854590 16326 tablet_bootstrap.cc:654] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:50.854950 16325 tablet_bootstrap.cc:492] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: No bootstrap required, opened a new log
I20260810 04:49:50.855001 16325 ts_tablet_manager.cc:1397] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:50.855095 16326 log.cc:824] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:50.855778 16326 tablet_bootstrap.cc:492] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: No bootstrap required, opened a new log
I20260810 04:49:50.855857 16326 ts_tablet_manager.cc:1397] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:49:50.856547 16325 raft_consensus.cc:348] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.856757 16325 raft_consensus.cc:374] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:50.856813 16325 raft_consensus.cc:729] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638cddcbf0734a628da45c6251d88db3, State: Initialized, Role: FOLLOWER
I20260810 04:49:50.856930 16325 consensus_queue.cc:260] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.857194 16325 ts_tablet_manager.cc:1428] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:50.857239 16298 heartbeater.cc:503] Master 127.14.237.62:44053 was elected leader, sending a full tablet report...
I20260810 04:49:50.857446 16326 raft_consensus.cc:348] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.857679 16326 raft_consensus.cc:374] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:50.857728 16326 raft_consensus.cc:729] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ac67c09a9246fe9f8e33b10b775398, State: Initialized, Role: FOLLOWER
I20260810 04:49:50.857837 16326 consensus_queue.cc:260] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.858134 16091 heartbeater.cc:503] Master 127.14.237.62:44053 was elected leader, sending a full tablet report...
I20260810 04:49:50.858134 16326 ts_tablet_manager.cc:1428] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:50.862282 16332 raft_consensus.cc:482] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:50.862347 16332 raft_consensus.cc:504] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.862576 16332 leader_election.cc:288] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 638cddcbf0734a628da45c6251d88db3 (127.14.237.3:40141), 997cecab74224ed98db66251b0c84a6a (127.14.237.1:42355)
I20260810 04:49:50.865939 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.862777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:39369 (local address 127.14.237.1:42355)
0810 04:49:50.862813 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.862815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.862956 (+ 141us) server_negotiation.cc:407] Connection header received
0810 04:49:50.863022 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.863023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.863032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.863087 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:50.863289 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.863978 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.865196 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.865589 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.865644 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.865650 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.865821 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.865879 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:49:50.866029 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.862777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:39209 (local address 127.14.237.3:40141)
0810 04:49:50.862918 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.862920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.862932 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:49:50.862948 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.862949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.862955 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.863015 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:50.863165 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.863878 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.865267 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.865641 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.865715 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.865722 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.865923 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.865963 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260810 04:49:50.865942 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.862734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42355 (local address 127.14.237.2:39369)
0810 04:49:50.862885 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.862894 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:50.862972 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:50.863090 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:50.863091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:50.863092 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:50.863171 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.863173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.864055 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.864056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.865115 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.865117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.865668 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.865669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.865740 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.865744 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:50.865810 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:49:50.865853 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":23,"thread_start_us":81,"threads_started":1}
I20260810 04:49:50.866041 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.862734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:40141 (local address 127.14.237.2:39209)
0810 04:49:50.862797 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.862809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:50.862848 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:50.863019 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:50.863019 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:50.863021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:50.863081 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.863082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.863953 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.863954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.865111 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:50.865114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.865732 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.865733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:50.865846 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.865849 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:50.865904 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:49:50.865941 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 04:49:50.866525 16228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e86794994d4aa78ad41351ce18e1c6" candidate_uuid: "f2ac67c09a9246fe9f8e33b10b775398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638cddcbf0734a628da45c6251d88db3" is_pre_election: true
I20260810 04:49:50.866655 16228 raft_consensus.cc:2454] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ac67c09a9246fe9f8e33b10b775398 in term 0.
I20260810 04:49:50.866679 15795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e86794994d4aa78ad41351ce18e1c6" candidate_uuid: "f2ac67c09a9246fe9f8e33b10b775398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997cecab74224ed98db66251b0c84a6a" is_pre_election: true
I20260810 04:49:50.866837 15795 raft_consensus.cc:2454] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ac67c09a9246fe9f8e33b10b775398 in term 0.
I20260810 04:49:50.866940 15906 leader_election.cc:302] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 638cddcbf0734a628da45c6251d88db3, f2ac67c09a9246fe9f8e33b10b775398; no voters:
I20260810 04:49:50.867061 16332 raft_consensus.cc:2781] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:49:50.867115 16332 raft_consensus.cc:482] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:49:50.867138 16332 raft_consensus.cc:3037] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:50.867736 16332 raft_consensus.cc:504] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.867872 16332 leader_election.cc:288] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [CANDIDATE]: Term 1 election: Requested vote from peers 638cddcbf0734a628da45c6251d88db3 (127.14.237.3:40141), 997cecab74224ed98db66251b0c84a6a (127.14.237.1:42355)
I20260810 04:49:50.868026 15795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e86794994d4aa78ad41351ce18e1c6" candidate_uuid: "f2ac67c09a9246fe9f8e33b10b775398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997cecab74224ed98db66251b0c84a6a"
I20260810 04:49:50.868093 15795 raft_consensus.cc:3037] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:50.868690 15795 raft_consensus.cc:2454] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ac67c09a9246fe9f8e33b10b775398 in term 1.
I20260810 04:49:50.868782 16228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e86794994d4aa78ad41351ce18e1c6" candidate_uuid: "f2ac67c09a9246fe9f8e33b10b775398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638cddcbf0734a628da45c6251d88db3"
I20260810 04:49:50.868875 16228 raft_consensus.cc:3037] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:50.868875 15907 leader_election.cc:302] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997cecab74224ed98db66251b0c84a6a, f2ac67c09a9246fe9f8e33b10b775398; no voters:
I20260810 04:49:50.868968 16332 raft_consensus.cc:2781] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:50.869130 16332 raft_consensus.cc:686] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 1 LEADER]: Becoming Leader. State: Replica: f2ac67c09a9246fe9f8e33b10b775398, State: Running, Role: LEADER
I20260810 04:49:50.869275 16332 consensus_queue.cc:237] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } }
I20260810 04:49:50.869741 16228 raft_consensus.cc:2454] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ac67c09a9246fe9f8e33b10b775398 in term 1.
I20260810 04:49:50.870102 15556 catalog_manager.cc:5495] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2ac67c09a9246fe9f8e33b10b775398 (127.14.237.2). New cstate: current_term: 1 leader_uuid: "f2ac67c09a9246fe9f8e33b10b775398" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2ac67c09a9246fe9f8e33b10b775398" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 42771 } health_report { overall_health: HEALTHY } } }
I20260810 04:49:50.881371 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.878988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.14.237.2:42771)
0810 04:49:50.879042 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:50.879044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:50.879052 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:49:50.879109 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:50.879110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:50.879126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:50.879200 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:49:50.879349 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.879777 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.880503 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:50.880783 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:50.880843 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:50.881009 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:49:50.881115 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:49:50.881204 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:50.881266 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 04:49:50.881286 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:49:50.883270 15795 raft_consensus.cc:1261] T 38e86794994d4aa78ad41351ce18e1c6 P 997cecab74224ed98db66251b0c84a6a [term 1 FOLLOWER]: Refusing update from remote peer f2ac67c09a9246fe9f8e33b10b775398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:50.883307 16228 raft_consensus.cc:1261] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 1 FOLLOWER]: Refusing update from remote peer f2ac67c09a9246fe9f8e33b10b775398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:50.883594 16332 consensus_queue.cc:1035] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997cecab74224ed98db66251b0c84a6a" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:50.883745 16334 consensus_queue.cc:1035] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "638cddcbf0734a628da45c6251d88db3" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:50.885932 16340 mvcc.cc:205] Tried to move back new op lower bound from 7316837953055195136 to 7316837953001373696. Current Snapshot: MvccSnapshot[applied={T|T < 7316837953055195136}]
I20260810 04:49:50.886730 16341 mvcc.cc:205] Tried to move back new op lower bound from 7316837953055195136 to 7316837953001373696. Current Snapshot: MvccSnapshot[applied={T|T < 7316837953055195136}]
I20260810 04:49:50.888316 16344 mvcc.cc:205] Tried to move back new op lower bound from 7316837953055195136 to 7316837953001373696. Current Snapshot: MvccSnapshot[applied={T|T < 7316837953055195136}]
I20260810 04:49:50.888969 15556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:50.889048 15556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:50.889807 15556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999abe69335849ee826b68e4d125e74b: Sending DeleteTablet for 3 replicas of tablet 38e86794994d4aa78ad41351ce18e1c6
I20260810 04:49:50.890125 16203 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e86794994d4aa78ad41351ce18e1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:50 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:35572
I20260810 04:49:50.890134 15982 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e86794994d4aa78ad41351ce18e1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:50 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:49336
I20260810 04:49:50.890159 15755 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e86794994d4aa78ad41351ce18e1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:50 UTC) from {username='kudu', principal='kudu/127.14.237.62@KRBTEST.COM'} at 127.0.0.1:37434
I20260810 04:49:50.890355 16355 tablet_replica.cc:330] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: stopping tablet replica
I20260810 04:49:50.890455 16355 raft_consensus.cc:2229] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:50.890519 16355 raft_consensus.cc:2258] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:50.890635 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 15655
I20260810 04:49:50.900519 16353 tablet_replica.cc:330] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: stopping tablet replica
I20260810 04:49:50.900648 16353 raft_consensus.cc:2229] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:49:50.900768 16353 raft_consensus.cc:2258] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:50.901654 16355 ts_tablet_manager.cc:1905] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:50.903347 16353 ts_tablet_manager.cc:1905] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:50.903388 16355 ts_tablet_manager.cc:1918] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:50.903568 16355 log.cc:1196] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/38e86794994d4aa78ad41351ce18e1c6
I20260810 04:49:50.903934 16355 ts_tablet_manager.cc:1939] T 38e86794994d4aa78ad41351ce18e1c6 P 638cddcbf0734a628da45c6251d88db3: Deleting consensus metadata
I20260810 04:49:50.904420 15537 catalog_manager.cc:4841] TS 638cddcbf0734a628da45c6251d88db3 (127.14.237.3:40141): tablet 38e86794994d4aa78ad41351ce18e1c6 (table test-table [id=27af13e53db744d8830ee5388cc624fc]) successfully deleted
I20260810 04:49:50.904945 16353 ts_tablet_manager.cc:1918] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:50.905036 16353 log.cc:1196] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/38e86794994d4aa78ad41351ce18e1c6
I20260810 04:49:50.905357 16353 ts_tablet_manager.cc:1939] T 38e86794994d4aa78ad41351ce18e1c6 P f2ac67c09a9246fe9f8e33b10b775398: Deleting consensus metadata
I20260810 04:49:50.905756 15535 catalog_manager.cc:4841] TS f2ac67c09a9246fe9f8e33b10b775398 (127.14.237.2:42771): tablet 38e86794994d4aa78ad41351ce18e1c6 (table test-table [id=27af13e53db744d8830ee5388cc624fc]) successfully deleted
W20260810 04:49:50.912511 15538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:42355 (error 108)
I20260810 04:49:50.913048 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:50.912899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:42355 (local address 127.0.0.1:37436)
0810 04:49:50.912963 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:50.913005 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:42355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 04:49:50.913146 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 15876
W20260810 04:49:50.917794 15538 catalog_manager.cc:4584] TS 997cecab74224ed98db66251b0c84a6a (127.14.237.1:42355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38e86794994d4aa78ad41351ce18e1c6: Network error: Client connection negotiation failed: client connection to 127.14.237.1:42355: connect: Connection refused (error 111)
I20260810 04:49:50.925448 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16097
I20260810 04:49:50.935550 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 15507
2026-08-10T04:49:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T04:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:50Z Disabled control of system clock
I20260810 04:49:50.968056 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38727
--webserver_interface=127.14.237.62
--webserver_port=0
--builtin_ntp_servers=127.14.237.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:38727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:51.048072 16364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:51.048281 16364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:51.048300 16364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:51.049453 16364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:51.049492 16364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:51.049505 16364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:51.049517 16364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:51.049530 16364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:51.050901 16364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:38727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:38727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:51.051141 16364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:51.051422 16364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:51.056298 16379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:51.056452 16364 server_base.cc:1034] running on GCE node
W20260810 04:49:51.056341 16378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:51.056373 16382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:51.056842 16364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:51.057122 16364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:51.058311 16364 hybrid_clock.cc:648] HybridClock initialized: now 1786337391058309 us; error 38 us; skew 500 ppm
I20260810 04:49:51.059680 16364 webserver.cc:460] Webserver started at http://127.14.237.62:45211/ using document root <none> and password file <none>
I20260810 04:49:51.059930 16364 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:51.059999 16364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:51.060158 16364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:51.061270 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "d875a4c641a84e2b9555faeddf862f50"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "75ee7f0419e37ba6ebbc1ea01d65a206"
server_key_iv: "cc16728d1809a73048ea91d0c2bb6ee1"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.061645 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "d875a4c641a84e2b9555faeddf862f50"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "75ee7f0419e37ba6ebbc1ea01d65a206"
server_key_iv: "cc16728d1809a73048ea91d0c2bb6ee1"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.062974 16364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:51.063832 16390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.064049 16364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:51.064183 16364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "d875a4c641a84e2b9555faeddf862f50"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "75ee7f0419e37ba6ebbc1ea01d65a206"
server_key_iv: "cc16728d1809a73048ea91d0c2bb6ee1"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.064268 16364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:51.082145 16364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:51.082417 16364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:51.082568 16364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:51.086915 16364 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:38727
I20260810 04:49:51.086953 16470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:38727 every 8 connection(s)
I20260810 04:49:51.087282 16364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:51.088236 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:51.090577 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50: Bootstrap starting.
I20260810 04:49:51.091197 16471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:51.091499 16471 log.cc:824] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:51.092185 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50: No bootstrap required, opened a new log
I20260810 04:49:51.092618 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16364
I20260810 04:49:51.092764 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:51.092950 15284 external_mini_cluster.cc:1428] Setting key 5fc4552e33c9518cc196348a374f882c
I20260810 04:49:51.094265 16471 raft_consensus.cc:348] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d875a4c641a84e2b9555faeddf862f50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38727 } }
I20260810 04:49:51.094419 16471 raft_consensus.cc:374] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:51.094465 16471 raft_consensus.cc:729] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d875a4c641a84e2b9555faeddf862f50, State: Initialized, Role: FOLLOWER
I20260810 04:49:51.094579 16471 consensus_queue.cc:260] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d875a4c641a84e2b9555faeddf862f50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38727 } }
I20260810 04:49:51.094667 16471 raft_consensus.cc:388] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:51.094709 16471 raft_consensus.cc:482] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:51.094766 16471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:51.095455 16471 raft_consensus.cc:504] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d875a4c641a84e2b9555faeddf862f50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38727 } }
I20260810 04:49:51.095566 16471 leader_election.cc:302] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d875a4c641a84e2b9555faeddf862f50; no voters:
I20260810 04:49:51.095731 16471 leader_election.cc:288] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:51.095796 16478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:51.095960 16471 sys_catalog.cc:564] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:51.095943 16478 raft_consensus.cc:686] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [term 1 LEADER]: Becoming Leader. State: Replica: d875a4c641a84e2b9555faeddf862f50, State: Running, Role: LEADER
I20260810 04:49:51.096091 16478 consensus_queue.cc:237] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d875a4c641a84e2b9555faeddf862f50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38727 } }
I20260810 04:49:51.096544 16478 sys_catalog.cc:455] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d875a4c641a84e2b9555faeddf862f50. Latest consensus state: current_term: 1 leader_uuid: "d875a4c641a84e2b9555faeddf862f50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d875a4c641a84e2b9555faeddf862f50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38727 } } }
I20260810 04:49:51.096647 16478 sys_catalog.cc:458] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:51.096555 16480 sys_catalog.cc:455] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d875a4c641a84e2b9555faeddf862f50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d875a4c641a84e2b9555faeddf862f50" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 38727 } } }
I20260810 04:49:51.096787 16480 sys_catalog.cc:458] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:51.096845 16489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:51.097426 16489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:51.098928 16489 catalog_manager.cc:1347] Generated new cluster ID: 26c048a0b73340e2b667b4d529539dfb
I20260810 04:49:51.098982 16489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:51.099944 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.093532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.14.237.62:38727)
0810 04:49:51.093740 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.093752 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.093777 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:49:51.094275 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.094283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.094346 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.094501 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:51.094796 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.096538 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.097756 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.098374 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.098495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.099250 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:51.099254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:51.099255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:51.099463 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:51.099504 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:51.099518 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:51.099589 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.099671 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.099746 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260810 04:49:51.128022 16489 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:51.128583 16489 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:51.130812 16489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50: Generated new TSK 0
I20260810 04:49:51.131014 16489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:49:51.136585 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.1:0
--local_ip_for_outbound_sockets=127.14.237.1
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38727
--builtin_ntp_servers=127.14.237.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:51.229203 16504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:51.229434 16504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:51.229454 16504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:51.230485 16504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:51.230520 16504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:51.230542 16504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.1
I20260810 04:49:51.231858 16504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.237.1
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.1
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:51.232124 16504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:51.232470 16504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:49:51.237432 16504 server_base.cc:1034] running on GCE node
W20260810 04:49:51.237401 16512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:51.237407 16515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:51.237535 16517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:51.237866 16504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:51.238065 16504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:51.239199 16504 hybrid_clock.cc:648] HybridClock initialized: now 1786337391239190 us; error 34 us; skew 500 ppm
I20260810 04:49:51.240420 16504 webserver.cc:460] Webserver started at http://127.14.237.1:39461/ using document root <none> and password file <none>
I20260810 04:49:51.240662 16504 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:51.240707 16504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:51.240844 16504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:51.241901 16504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/data/instance:
uuid: "10d58e27a29245d59f2afd4d5724c3a5"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "07e6ee7bb7f1acabe4b830108f34f0ea"
server_key_iv: "08a1c07021ca2543535ea9ea4ee82fc3"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.242237 16504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance:
uuid: "10d58e27a29245d59f2afd4d5724c3a5"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "07e6ee7bb7f1acabe4b830108f34f0ea"
server_key_iv: "08a1c07021ca2543535ea9ea4ee82fc3"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.243611 16504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:49:51.244349 16528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.244539 16504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:51.244634 16504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/wal
uuid: "10d58e27a29245d59f2afd4d5724c3a5"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "07e6ee7bb7f1acabe4b830108f34f0ea"
server_key_iv: "08a1c07021ca2543535ea9ea4ee82fc3"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.244701 16504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:51.257978 16504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:51.258255 16504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:51.258394 16504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:51.258594 16504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:51.258926 16504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:51.258983 16504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.259023 16504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:51.259052 16504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.264962 16504 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.1:33777
I20260810 04:49:51.265008 16706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.1:33777 every 8 connection(s)
I20260810 04:49:51.265367 16504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/data/info.pb
I20260810 04:49:51.270726 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.266141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.1:59213 (local address 127.14.237.62:38727)
0810 04:49:51.266220 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.266223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.266439 (+ 216us) server_negotiation.cc:407] Connection header received
0810 04:49:51.266960 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.266961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.266976 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.267041 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:51.267737 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.268259 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.269305 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.269533 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.269634 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.270474 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:51.270475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:51.270475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:51.270480 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:51.270493 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:51.270497 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:51.270543 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.270622 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.270651 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:49:51.270926 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.266101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38727 (local address 127.14.237.1:59213)
0810 04:49:51.266316 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.266348 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:51.266849 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:51.267120 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:51.267124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:51.267134 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:51.267633 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.267637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.268361 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.268363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.269162 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.269164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.269651 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.269653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.269782 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.270319 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:51.270320 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:51.270346 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:51.270368 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:51.270543 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:51.270549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:51.270552 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:51.270618 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:49:51.270714 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260810 04:49:51.271540 16707 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38727
I20260810 04:49:51.271620 16707 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:51.271751 16707 heartbeater.cc:511] Master 127.14.237.62:38727 requested a full tablet report, sending...
I20260810 04:49:51.272228 16414 ts_manager.cc:194] Registered new tserver with Master: 10d58e27a29245d59f2afd4d5724c3a5 (127.14.237.1:33777)
I20260810 04:49:51.272773 16414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.1:59213
I20260810 04:49:51.272979 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16504
I20260810 04:49:51.273103 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-0/wal/instance
I20260810 04:49:51.273340 15284 external_mini_cluster.cc:1428] Setting key 2dccc4519ddb8681ce921a3aa51edac0
I20260810 04:49:51.274194 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.2:0
--local_ip_for_outbound_sockets=127.14.237.2
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38727
--builtin_ntp_servers=127.14.237.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:51.355743 16715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:51.355937 16715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:51.355959 16715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:51.357084 16715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:51.357125 16715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:51.357151 16715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.2
I20260810 04:49:51.358619 16715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.237.2
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.2
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:51.358856 16715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:51.359166 16715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:51.364306 16732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:51.364288 16727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:51.364274 16735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:51.364565 16715 server_base.cc:1034] running on GCE node
I20260810 04:49:51.364799 16715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:51.365033 16715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:51.366223 16715 hybrid_clock.cc:648] HybridClock initialized: now 1786337391366206 us; error 28 us; skew 500 ppm
I20260810 04:49:51.367449 16715 webserver.cc:460] Webserver started at http://127.14.237.2:39359/ using document root <none> and password file <none>
I20260810 04:49:51.367687 16715 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:51.367756 16715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:51.367900 16715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:51.369046 16715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/data/instance:
uuid: "5f65469f77be4f1cb281266ce4586afd"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "d50b4228f1294487489a82e597339b3d"
server_key_iv: "2bc3496de8bf42c01530db3f9f532e71"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.369450 16715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance:
uuid: "5f65469f77be4f1cb281266ce4586afd"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "d50b4228f1294487489a82e597339b3d"
server_key_iv: "2bc3496de8bf42c01530db3f9f532e71"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.370721 16715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:51.371399 16746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.371578 16715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:51.371670 16715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/wal
uuid: "5f65469f77be4f1cb281266ce4586afd"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "d50b4228f1294487489a82e597339b3d"
server_key_iv: "2bc3496de8bf42c01530db3f9f532e71"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.371737 16715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:51.396651 16715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:51.396996 16715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:51.397163 16715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:51.397430 16715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:51.397782 16715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:51.397817 16715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.397840 16715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:51.397889 16715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.403417 16715 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.2:46735
I20260810 04:49:51.403496 16950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.2:46735 every 8 connection(s)
I20260810 04:49:51.403844 16715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/data/info.pb
I20260810 04:49:51.409296 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.404603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:45857 (local address 127.14.237.62:38727)
0810 04:49:51.404676 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.404678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.404938 (+ 260us) server_negotiation.cc:407] Connection header received
0810 04:49:51.405552 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.405554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.405566 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.405643 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:51.406369 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.406882 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.407866 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.408079 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.408162 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.408960 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:51.408961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:51.408962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:51.408966 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:51.408980 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:51.408984 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:51.409040 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.409171 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.409200 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:49:51.409540 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.404582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38727 (local address 127.14.237.2:45857)
0810 04:49:51.404796 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.404830 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:51.405431 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:51.405722 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:51.405725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:51.405734 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:51.406236 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.406240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.406988 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.406989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.407771 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.407773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.408174 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.408175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.408302 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.408814 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:51.408815 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:51.408843 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:51.408866 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:51.409042 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:51.409046 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:51.409050 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:51.409171 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 04:49:51.409314 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260810 04:49:51.409942 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16715
I20260810 04:49:51.410068 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/wal/instance
I20260810 04:49:51.410087 16951 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38727
I20260810 04:49:51.410185 16951 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:51.410291 15284 external_mini_cluster.cc:1428] Setting key ff216802db036ead62b0a8cfbd19b117
I20260810 04:49:51.410400 16951 heartbeater.cc:511] Master 127.14.237.62:38727 requested a full tablet report, sending...
I20260810 04:49:51.410810 16413 ts_manager.cc:194] Registered new tserver with Master: 5f65469f77be4f1cb281266ce4586afd (127.14.237.2:46735)
I20260810 04:49:51.411212 16413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.2:45857
I20260810 04:49:51.411250 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.237.3:0
--local_ip_for_outbound_sockets=127.14.237.3
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38727
--builtin_ntp_servers=127.14.237.20:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:49:51.492295 16956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:51.492506 16956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:51.492528 16956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:51.493719 16956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:51.493759 16956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:51.493784 16956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.237.3
I20260810 04:49:51.495131 16956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.237.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.237.3
--webserver_port=0
--tserver_master_addrs=127.14.237.62:38727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.237.3
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:51.495371 16956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:51.495687 16956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:51.500643 16969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:51.500671 16971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:51.500684 16974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:51.500905 16956 server_base.cc:1034] running on GCE node
I20260810 04:49:51.501108 16956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:51.501446 16956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:51.502590 16956 hybrid_clock.cc:648] HybridClock initialized: now 1786337391502574 us; error 34 us; skew 500 ppm
I20260810 04:49:51.503836 16956 webserver.cc:460] Webserver started at http://127.14.237.3:43801/ using document root <none> and password file <none>
I20260810 04:49:51.504091 16956 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:51.504138 16956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:51.504308 16956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:51.505396 16956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/data/instance:
uuid: "3eaddecd0de54f22a127b9cd3ec07820"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "b0e8e4ec62c979eadfa70550bdaca966"
server_key_iv: "d9a8397ec06d666fd6d52dc3953d09c6"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.505748 16956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance:
uuid: "3eaddecd0de54f22a127b9cd3ec07820"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "b0e8e4ec62c979eadfa70550bdaca966"
server_key_iv: "d9a8397ec06d666fd6d52dc3953d09c6"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.507012 16956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:51.507683 16983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.507874 16956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:51.507992 16956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/wal
uuid: "3eaddecd0de54f22a127b9cd3ec07820"
format_stamp: "Formatted at 2026-08-10 04:49:51 on dist-test-slave-8hhm"
server_key: "b0e8e4ec62c979eadfa70550bdaca966"
server_key_iv: "d9a8397ec06d666fd6d52dc3953d09c6"
server_key_version: "encryptionkey@0"
I20260810 04:49:51.508061 16956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:51.529714 16956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:51.530019 16956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:51.530181 16956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:51.530405 16956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:49:51.530759 16956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:49:51.530814 16956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.530856 16956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:49:51.530882 16956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:51.536522 16956 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.3:43237
I20260810 04:49:51.536571 17171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.3:43237 every 8 connection(s)
I20260810 04:49:51.536866 16956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/data/info.pb
I20260810 04:49:51.541939 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.537657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.3:43793 (local address 127.14.237.62:38727)
0810 04:49:51.537747 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.537749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.537943 (+ 194us) server_negotiation.cc:407] Connection header received
0810 04:49:51.538483 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.538484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.538498 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.538545 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:51.539263 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.539756 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.540678 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.540882 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.540961 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.541672 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:51.541673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:51.541673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:51.541678 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:51.541691 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:51.541695 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:51.541746 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.541834 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.541864 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:49:51.542171 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.537603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:38727 (local address 127.14.237.3:43793)
0810 04:49:51.537822 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.537856 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:51.538371 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:51.538636 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:51.538639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:51.538647 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:49:51.539163 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.539168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.539855 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.539856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.540586 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.540588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.540969 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.540971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.541096 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.541538 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:49:51.541538 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:49:51.541564 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:49:51.541585 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:49:51.541749 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:49:51.541753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:49:51.541756 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:49:51.541835 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:49:51.541926 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260810 04:49:51.542654 17174 heartbeater.cc:347] Connected to a master server at 127.14.237.62:38727
I20260810 04:49:51.542745 17174 heartbeater.cc:464] Registering TS with master...
I20260810 04:49:51.542899 17174 heartbeater.cc:511] Master 127.14.237.62:38727 requested a full tablet report, sending...
I20260810 04:49:51.543236 16413 ts_manager.cc:194] Registered new tserver with Master: 3eaddecd0de54f22a127b9cd3ec07820 (127.14.237.3:43237)
I20260810 04:49:51.543602 16413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.237.3:43793
I20260810 04:49:51.546826 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 16956
I20260810 04:49:51.546928 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/wal/instance
I20260810 04:49:51.547178 15284 external_mini_cluster.cc:1428] Setting key 9ac2cec648e353c0f58d2f7a9786834c
I20260810 04:49:51.548122 15284 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:49:51.552224 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.549617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.14.237.62:38727)
0810 04:49:51.549661 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.549663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.549783 (+ 120us) server_negotiation.cc:407] Connection header received
0810 04:49:51.549827 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.549827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.549837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.549881 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:51.549992 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.550438 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.551181 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.551358 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.551414 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.551987 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:51.551988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:51.551988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:51.551992 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:51.552003 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:51.552006 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:51.552048 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.552102 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.552121 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:49:51.553999 16413 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:49:51.555495 16413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:49:51.562005 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.557534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:33777 (local address 127.0.0.1:37270)
0810 04:49:51.557662 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.557700 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:51.557761 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:51.558203 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:51.558206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:51.558209 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:51.558381 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.558383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.559775 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.559776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.560864 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.560867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.561784 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.561785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.561866 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.561876 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 04:49:51.561925 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:49:51.561958 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260810 04:49:51.562032 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.557713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.14.237.1:33777)
0810 04:49:51.557960 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.557964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.557975 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:49:51.558018 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.558019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.558051 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.558192 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:51.558467 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.559651 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.560954 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.561682 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.561768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.561854 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.561944 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.561971 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260810 04:49:51.562322 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.557567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.2:46735 (local address 127.0.0.1:44054)
0810 04:49:51.557895 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.557904 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:51.557942 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:51.558448 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:51.558448 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:51.558451 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:51.558509 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.558511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.559480 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.559480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.560865 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.560867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.562113 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.562113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.562174 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.562178 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:51.562229 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:49:51.562263 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":28,"thread_start_us":78,"threads_started":1}
I20260810 04:49:51.562363 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.557926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.14.237.2:46735)
0810 04:49:51.558283 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.558287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.558295 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:49:51.558333 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.558335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.558349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.558588 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:51.558654 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.559390 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.560954 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.561682 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.561768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.561854 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.562246 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.562284 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":98,"threads_started":1}
I20260810 04:49:51.562937 16835 tablet_service.cc:1467] Processing CreateTablet for tablet 809107150dea4dd29f7c0588febebe60 (DEFAULT_TABLE table=test-table [id=bac0c072b4874f7ba5c59e544e650ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:51.562937 16610 tablet_service.cc:1467] Processing CreateTablet for tablet 809107150dea4dd29f7c0588febebe60 (DEFAULT_TABLE table=test-table [id=bac0c072b4874f7ba5c59e544e650ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:51.563298 16835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809107150dea4dd29f7c0588febebe60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:51.563298 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809107150dea4dd29f7c0588febebe60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:51.566304 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.557761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:43237 (local address 127.0.0.1:43072)
0810 04:49:51.558136 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.558146 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:51.558204 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:51.558450 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:51.558450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:51.558452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:51.558495 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.558496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.560964 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.560965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.561696 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.561697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.566068 (+ 4371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.566069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.566130 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.566134 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:51.566191 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:49:51.566239 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260810 04:49:51.566318 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.557829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.14.237.3:43237)
0810 04:49:51.558049 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.558054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.558272 (+ 218us) server_negotiation.cc:407] Connection header received
0810 04:49:51.558309 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.558311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.558326 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.558394 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:51.558567 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.559673 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.565290 (+ 5617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.565966 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.566053 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.566133 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.566204 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.566238 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260810 04:49:51.567018 17203 tablet_bootstrap.cc:492] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5: Bootstrap starting.
I20260810 04:49:51.567149 17198 tablet_bootstrap.cc:492] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Bootstrap starting.
I20260810 04:49:51.567189 17071 tablet_service.cc:1467] Processing CreateTablet for tablet 809107150dea4dd29f7c0588febebe60 (DEFAULT_TABLE table=test-table [id=bac0c072b4874f7ba5c59e544e650ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:49:51.567565 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809107150dea4dd29f7c0588febebe60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:51.567799 17203 tablet_bootstrap.cc:654] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:51.567830 17198 tablet_bootstrap.cc:654] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:51.568109 17203 log.cc:824] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:51.568146 17198 log.cc:824] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:51.568915 17203 tablet_bootstrap.cc:492] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5: No bootstrap required, opened a new log
I20260810 04:49:51.568991 17203 ts_tablet_manager.cc:1397] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:49:51.569031 17198 tablet_bootstrap.cc:492] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: No bootstrap required, opened a new log
I20260810 04:49:51.569110 17198 ts_tablet_manager.cc:1397] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:49:51.570974 17210 tablet_bootstrap.cc:492] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Bootstrap starting.
I20260810 04:49:51.570990 17198 raft_consensus.cc:348] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.571148 17198 raft_consensus.cc:374] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:51.571197 17198 raft_consensus.cc:729] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f65469f77be4f1cb281266ce4586afd, State: Initialized, Role: FOLLOWER
I20260810 04:49:51.571331 17198 consensus_queue.cc:260] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.571614 17198 ts_tablet_manager.cc:1428] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:49:51.571663 16951 heartbeater.cc:503] Master 127.14.237.62:38727 was elected leader, sending a full tablet report...
I20260810 04:49:51.571593 17203 raft_consensus.cc:348] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.571755 17203 raft_consensus.cc:374] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:51.571815 17203 raft_consensus.cc:729] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d58e27a29245d59f2afd4d5724c3a5, State: Initialized, Role: FOLLOWER
I20260810 04:49:51.571905 17210 tablet_bootstrap.cc:654] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:51.571951 17203 consensus_queue.cc:260] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.572234 17210 log.cc:824] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:51.572237 17203 ts_tablet_manager.cc:1428] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:49:51.572279 16707 heartbeater.cc:503] Master 127.14.237.62:38727 was elected leader, sending a full tablet report...
I20260810 04:49:51.573094 17210 tablet_bootstrap.cc:492] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: No bootstrap required, opened a new log
I20260810 04:49:51.573186 17210 ts_tablet_manager.cc:1397] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:49:51.575058 17210 raft_consensus.cc:348] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.575176 17210 raft_consensus.cc:374] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:51.575196 17210 raft_consensus.cc:729] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eaddecd0de54f22a127b9cd3ec07820, State: Initialized, Role: FOLLOWER
I20260810 04:49:51.575335 17210 consensus_queue.cc:260] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.575591 17210 ts_tablet_manager.cc:1428] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:49:51.575634 17174 heartbeater.cc:503] Master 127.14.237.62:38727 was elected leader, sending a full tablet report...
W20260810 04:49:51.654502 16952 tablet.cc:2369] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:49:51.765978 16711 tablet.cc:2369] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:49:51.787469 17179 tablet.cc:2369] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:49:51.799190 17211 raft_consensus.cc:482] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:49:51.799300 17211 raft_consensus.cc:504] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.799692 17211 leader_election.cc:288] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10d58e27a29245d59f2afd4d5724c3a5 (127.14.237.1:33777), 3eaddecd0de54f22a127b9cd3ec07820 (127.14.237.3:43237)
I20260810 04:49:51.803701 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.800089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:45603 (local address 127.14.237.3:43237)
0810 04:49:51.800132 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.800134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.800275 (+ 141us) server_negotiation.cc:407] Connection header received
0810 04:49:51.800306 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.800306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.800316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.800379 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:51.800512 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.801259 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.802809 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.803341 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.803418 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.803426 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.803596 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.803632 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:49:51.803701 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.800101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.237.2:32865 (local address 127.14.237.1:33777)
0810 04:49:51.800161 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.800163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.800171 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:49:51.800199 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.800200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.800217 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.800318 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:49:51.800512 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.801259 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.802809 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.803341 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.803418 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.803426 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.803596 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.803632 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:49:51.803728 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.799882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:33777 (local address 127.14.237.2:32865)
0810 04:49:51.799944 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.799956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:51.800001 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:51.800338 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:51.800339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:51.800341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:51.800420 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.800422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.801333 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.801334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.802699 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.802702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.803436 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.803438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.803511 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.803515 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:49:51.803575 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:49:51.803619 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":90}
I20260810 04:49:51.803736 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.800007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.3:43237 (local address 127.14.237.2:45603)
0810 04:49:51.800190 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.800199 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:49:51.800257 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:49:51.800387 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:49:51.800388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:49:51.800389 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:49:51.800445 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.800446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.801333 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.801334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.802699 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:49:51.802702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.803448 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.803449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:49:51.803511 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.803516 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:49:51.803583 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 04:49:51.803617 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":44,"thread_start_us":137,"threads_started":1}
I20260810 04:49:51.804051 16638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809107150dea4dd29f7c0588febebe60" candidate_uuid: "5f65469f77be4f1cb281266ce4586afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d58e27a29245d59f2afd4d5724c3a5" is_pre_election: true
I20260810 04:49:51.804209 16638 raft_consensus.cc:2454] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f65469f77be4f1cb281266ce4586afd in term 0.
I20260810 04:49:51.804172 17103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809107150dea4dd29f7c0588febebe60" candidate_uuid: "5f65469f77be4f1cb281266ce4586afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eaddecd0de54f22a127b9cd3ec07820" is_pre_election: true
I20260810 04:49:51.804335 17103 raft_consensus.cc:2454] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f65469f77be4f1cb281266ce4586afd in term 0.
I20260810 04:49:51.804457 16754 leader_election.cc:302] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d58e27a29245d59f2afd4d5724c3a5, 5f65469f77be4f1cb281266ce4586afd; no voters:
I20260810 04:49:51.804595 17211 raft_consensus.cc:2781] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:49:51.804649 17211 raft_consensus.cc:482] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:49:51.804665 17211 raft_consensus.cc:3037] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:51.805430 17211 raft_consensus.cc:504] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.805553 17211 leader_election.cc:288] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [CANDIDATE]: Term 1 election: Requested vote from peers 10d58e27a29245d59f2afd4d5724c3a5 (127.14.237.1:33777), 3eaddecd0de54f22a127b9cd3ec07820 (127.14.237.3:43237)
I20260810 04:49:51.805749 16638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809107150dea4dd29f7c0588febebe60" candidate_uuid: "5f65469f77be4f1cb281266ce4586afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d58e27a29245d59f2afd4d5724c3a5"
I20260810 04:49:51.805799 17103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809107150dea4dd29f7c0588febebe60" candidate_uuid: "5f65469f77be4f1cb281266ce4586afd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eaddecd0de54f22a127b9cd3ec07820"
I20260810 04:49:51.805845 16638 raft_consensus.cc:3037] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:51.805876 17103 raft_consensus.cc:3037] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:51.806563 16638 raft_consensus.cc:2454] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f65469f77be4f1cb281266ce4586afd in term 1.
I20260810 04:49:51.806588 17103 raft_consensus.cc:2454] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f65469f77be4f1cb281266ce4586afd in term 1.
I20260810 04:49:51.806735 16754 leader_election.cc:302] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eaddecd0de54f22a127b9cd3ec07820, 5f65469f77be4f1cb281266ce4586afd; no voters:
I20260810 04:49:51.806878 17211 raft_consensus.cc:2781] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:51.807045 17211 raft_consensus.cc:686] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 1 LEADER]: Becoming Leader. State: Replica: 5f65469f77be4f1cb281266ce4586afd, State: Running, Role: LEADER
I20260810 04:49:51.807169 17211 consensus_queue.cc:237] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } }
I20260810 04:49:51.808041 16414 catalog_manager.cc:5495] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f65469f77be4f1cb281266ce4586afd (127.14.237.2). New cstate: current_term: 1 leader_uuid: "5f65469f77be4f1cb281266ce4586afd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f65469f77be4f1cb281266ce4586afd" member_type: VOTER last_known_addr { host: "127.14.237.2" port: 46735 } health_report { overall_health: HEALTHY } } }
I20260810 04:49:51.842097 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.840062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.14.237.2:46735)
0810 04:49:51.840126 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:51.840128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:51.840136 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:49:51.840203 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:51.840204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:51.840222 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:51.840276 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:49:51.840377 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.840815 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.841512 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:51.841679 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:51.841730 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:51.841834 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:49:51.841920 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:49:51.841970 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:51.842015 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 04:49:51.842029 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:49:51.843863 16638 raft_consensus.cc:1261] T 809107150dea4dd29f7c0588febebe60 P 10d58e27a29245d59f2afd4d5724c3a5 [term 1 FOLLOWER]: Refusing update from remote peer 5f65469f77be4f1cb281266ce4586afd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:51.843948 17103 raft_consensus.cc:1261] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 1 FOLLOWER]: Refusing update from remote peer 5f65469f77be4f1cb281266ce4586afd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:49:51.844321 17211 consensus_queue.cc:1035] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d58e27a29245d59f2afd4d5724c3a5" member_type: VOTER last_known_addr { host: "127.14.237.1" port: 33777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:51.844475 17223 consensus_queue.cc:1035] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eaddecd0de54f22a127b9cd3ec07820" member_type: VOTER last_known_addr { host: "127.14.237.3" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:49:51.847832 17232 mvcc.cc:205] Tried to move back new op lower bound from 7316837956989501440 to 7316837956842745856. Current Snapshot: MvccSnapshot[applied={T|T < 7316837956989501440}]
I20260810 04:49:51.849690 16413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:51.849777 16413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:49:51.850606 16413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d875a4c641a84e2b9555faeddf862f50: Sending DeleteTablet for 3 replicas of tablet 809107150dea4dd29f7c0588febebe60
I20260810 04:49:51.850997 17071 tablet_service.cc:1514] Processing DeleteTablet for tablet 809107150dea4dd29f7c0588febebe60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:51 UTC) from {username='slave'} at 127.0.0.1:43072
I20260810 04:49:51.851002 16835 tablet_service.cc:1514] Processing DeleteTablet for tablet 809107150dea4dd29f7c0588febebe60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:51 UTC) from {username='slave'} at 127.0.0.1:44054
I20260810 04:49:51.851037 16610 tablet_service.cc:1514] Processing DeleteTablet for tablet 809107150dea4dd29f7c0588febebe60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:49:51 UTC) from {username='slave'} at 127.0.0.1:37270
I20260810 04:49:51.851265 17247 tablet_replica.cc:330] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: stopping tablet replica
I20260810 04:49:51.851392 17247 raft_consensus.cc:2229] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:49:51.851511 17247 raft_consensus.cc:2258] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:51.851502 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16504
I20260810 04:49:51.851689 17250 tablet_replica.cc:330] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: stopping tablet replica
I20260810 04:49:51.851821 17250 raft_consensus.cc:2229] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:49:51.851946 17250 raft_consensus.cc:2258] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:49:51.852562 17247 ts_tablet_manager.cc:1905] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:51.854983 17247 ts_tablet_manager.cc:1918] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:51.855062 17247 log.cc:1196] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-2/wal/wals/809107150dea4dd29f7c0588febebe60
I20260810 04:49:51.855283 17247 ts_tablet_manager.cc:1939] T 809107150dea4dd29f7c0588febebe60 P 3eaddecd0de54f22a127b9cd3ec07820: Deleting consensus metadata
I20260810 04:49:51.855647 16396 catalog_manager.cc:4841] TS 3eaddecd0de54f22a127b9cd3ec07820 (127.14.237.3:43237): tablet 809107150dea4dd29f7c0588febebe60 (table test-table [id=bac0c072b4874f7ba5c59e544e650ffa]) successfully deleted
I20260810 04:49:51.857335 17250 ts_tablet_manager.cc:1905] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:49:51.859054 17250 ts_tablet_manager.cc:1918] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:49:51.859122 17250 log.cc:1196] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Deleting WAL directory at /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786337145367972-15284-0/minicluster-data/ts-1/wal/wals/809107150dea4dd29f7c0588febebe60
I20260810 04:49:51.859426 17250 ts_tablet_manager.cc:1939] T 809107150dea4dd29f7c0588febebe60 P 5f65469f77be4f1cb281266ce4586afd: Deleting consensus metadata
I20260810 04:49:51.859835 16393 catalog_manager.cc:4841] TS 5f65469f77be4f1cb281266ce4586afd (127.14.237.2:46735): tablet 809107150dea4dd29f7c0588febebe60 (table test-table [id=bac0c072b4874f7ba5c59e544e650ffa]) successfully deleted
W20260810 04:49:51.868273 16396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.237.1:33777 (error 108)
I20260810 04:49:51.868546 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16715
I20260810 04:49:51.870330 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:51.868675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.1:33777 (local address 127.0.0.1:37276)
0810 04:49:51.870235 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0810 04:49:51.870289 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1530}
W20260810 04:49:51.870517 16396 catalog_manager.cc:4584] TS 10d58e27a29245d59f2afd4d5724c3a5 (127.14.237.1:33777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 809107150dea4dd29f7c0588febebe60: Network error: Client connection negotiation failed: client connection to 127.14.237.1:33777: connect: Connection refused (error 111)
I20260810 04:49:51.883677 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16956
I20260810 04:49:51.895774 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 16364
2026-08-10T04:49:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (958 ms)
[----------] 6 tests from EncryptionPolicyTest (9336 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:51 dist-test-slave-8hhm krb5kdc[17257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:51 dist-test-slave-8hhm krb5kdc[17257](info): set up 2 sockets
Aug 10 04:49:51 dist-test-slave-8hhm krb5kdc[17257](info): commencing operation
krb5kdc: starting...
W20260810 04:49:53.150041 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:49:53 dist-test-slave-8hhm krb5kdc[17257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:49:53.264437 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:53.345579 17281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:53.345764 17281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:53.345785 17281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:53.346913 17281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:53.346956 17281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:53.346969 17281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:53.346982 17281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:53.347033 17281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:53.348402 17281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:53.348639 17281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:53.348865 17281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:53.353601 17289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:53.353631 17287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:53.353776 17281 server_base.cc:1034] running on GCE node
W20260810 04:49:53.353644 17286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:53.354041 17281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:53.354395 17281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:53.355564 17281 hybrid_clock.cc:648] HybridClock initialized: now 1786337393355544 us; error 31 us; skew 500 ppm
Aug 10 04:49:53 dist-test-slave-8hhm krb5kdc[17257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:53.362488 17281 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:49:53.362867 17281 webserver.cc:460] Webserver started at http://127.0.0.1:44127/ using document root <none> and password file <none>
I20260810 04:49:53.363081 17281 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:53.363173 17281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:53.363284 17281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:53.364250 17281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "50611422602f462d94bec586e8beaf12"
format_stamp: "Formatted at 2026-08-10 04:49:53 on dist-test-slave-8hhm"
server_key: "2ca15ca1bc10fb892a2e5e3f5672b18a"
server_key_iv: "b322a6aae769bb61e8d36681f412b941"
server_key_version: "encryptionkey@0"
I20260810 04:49:53.364550 17281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "50611422602f462d94bec586e8beaf12"
format_stamp: "Formatted at 2026-08-10 04:49:53 on dist-test-slave-8hhm"
server_key: "2ca15ca1bc10fb892a2e5e3f5672b18a"
server_key_iv: "b322a6aae769bb61e8d36681f412b941"
server_key_version: "encryptionkey@0"
I20260810 04:49:53.365916 17281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:49:53.366596 17298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:53.366834 17281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:49:53.366922 17281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "50611422602f462d94bec586e8beaf12"
format_stamp: "Formatted at 2026-08-10 04:49:53 on dist-test-slave-8hhm"
server_key: "2ca15ca1bc10fb892a2e5e3f5672b18a"
server_key_iv: "b322a6aae769bb61e8d36681f412b941"
server_key_version: "encryptionkey@0"
I20260810 04:49:53.366978 17281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:53.403209 17281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:53.405557 17281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:53.405668 17281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:53.409579 17281 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37281
I20260810 04:49:53.409639 17388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37281 every 8 connection(s)
I20260810 04:49:53.409998 17281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:53.410892 17389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:53.413158 17389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12: Bootstrap starting.
I20260810 04:49:53.413717 17389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:53.413951 17389 log.cc:824] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:53.414624 17389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12: No bootstrap required, opened a new log
I20260810 04:49:53.416090 17389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50611422602f462d94bec586e8beaf12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } }
I20260810 04:49:53.416224 17389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:53.416247 17389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50611422602f462d94bec586e8beaf12, State: Initialized, Role: FOLLOWER
I20260810 04:49:53.416342 17389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50611422602f462d94bec586e8beaf12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } }
I20260810 04:49:53.416424 17389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:53.416460 17389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:53.416527 17389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:53.417042 17389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50611422602f462d94bec586e8beaf12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } }
I20260810 04:49:53.417165 17389 leader_election.cc:302] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50611422602f462d94bec586e8beaf12; no voters:
I20260810 04:49:53.417378 17389 leader_election.cc:288] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:53.417419 17394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:53.417598 17394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [term 1 LEADER]: Becoming Leader. State: Replica: 50611422602f462d94bec586e8beaf12, State: Running, Role: LEADER
I20260810 04:49:53.417672 17389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:53.417728 17394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50611422602f462d94bec586e8beaf12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } }
I20260810 04:49:53.418115 17396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50611422602f462d94bec586e8beaf12. Latest consensus state: current_term: 1 leader_uuid: "50611422602f462d94bec586e8beaf12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50611422602f462d94bec586e8beaf12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } } }
I20260810 04:49:53.418215 17396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:53.418248 17395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50611422602f462d94bec586e8beaf12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50611422602f462d94bec586e8beaf12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } } }
I20260810 04:49:53.418314 17395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:53.418435 17403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:53.418983 17403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:53.419800 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 17281
I20260810 04:49:53.419922 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:53.420138 15284 external_mini_cluster.cc:1428] Setting key 068b768b963ad1a3000474157c589ba0
I20260810 04:49:53.420454 17403 catalog_manager.cc:1347] Generated new cluster ID: 483c235159e3490185de657316685dc8
I20260810 04:49:53.420521 17403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:49:53 dist-test-slave-8hhm krb5kdc[17257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:49:53.431281 17403 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:53.431627 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:53.420732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43349 (local address 127.0.0.1:37281)
0810 04:49:53.420912 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:53.420920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:53.420970 (+ 50us) server_negotiation.cc:407] Connection header received
0810 04:49:53.421091 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:53.421098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:53.421159 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:53.421341 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:53.422061 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:53.423487 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:53.424492 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:53.425125 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:53.425259 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:53.427938 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:53.427942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:53.427943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:53.427954 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:53.429841 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:53.430298 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:53.430300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:53.430301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:53.430401 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:53.430652 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:53.430653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:53.430653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:53.430918 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:53.431196 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:53.431281 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 04:49:53.431455 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260810 04:49:53.431962 17403 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:53.432538 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:49:53.440073 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:53.433943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38587 (local address 127.0.0.1:37281)
0810 04:49:53.433981 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:53.433983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:53.434108 (+ 125us) server_negotiation.cc:407] Connection header received
0810 04:49:53.434193 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:53.434194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:53.434203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:53.434264 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:53.434769 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:53.435234 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:53.435956 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:53.436180 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:53.436258 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:53.437440 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:53.437441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:53.437441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:53.437448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:53.438879 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:53.439267 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:53.439268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:53.439269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:53.439343 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:53.439568 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:53.439569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:53.439569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:53.439686 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:53.439793 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:53.439916 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 04:49:53.439975 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:49:53.444353 17403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50611422602f462d94bec586e8beaf12: Generated new TSK 0
I20260810 04:49:53.444510 17403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:49:53.447430 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 17281
W20260810 04:49:53.465307 17254 connection.cc:535] client connection to 127.0.0.1:37281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37281): Connection reset by peer (error 104)
2026-08-10T04:49:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1560 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:53 dist-test-slave-8hhm krb5kdc[17437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:53 dist-test-slave-8hhm krb5kdc[17437](info): set up 2 sockets
Aug 10 04:49:53 dist-test-slave-8hhm krb5kdc[17437](info): commencing operation
krb5kdc: starting...
W20260810 04:49:54.699151 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:49:54 dist-test-slave-8hhm krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:49:54.820147 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:54.906499 17469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:54.906699 17469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:54.906741 17469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:54.907882 17469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:54.907917 17469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:54.907931 17469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:54.907944 17469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:54.907956 17469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:54.909448 17469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:54.909642 17469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:54.909808 17469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:54.914485 17481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:54.914506 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:54.914507 17483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:54.914836 17469 server_base.cc:1034] running on GCE node
I20260810 04:49:54.915046 17469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:54.915324 17469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:54.916522 17469 hybrid_clock.cc:648] HybridClock initialized: now 1786337394916508 us; error 27 us; skew 500 ppm
Aug 10 04:49:54 dist-test-slave-8hhm krb5kdc[17437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:54.919236 17469 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:49:54.919787 17469 webserver.cc:460] Webserver started at http://127.0.0.1:41901/ using document root <none> and password file <none>
I20260810 04:49:54.920050 17469 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:54.920135 17469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:54.920292 17469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:54.921478 17469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "cb5873f02f0149cdb8bb9be2686e84cb"
format_stamp: "Formatted at 2026-08-10 04:49:54 on dist-test-slave-8hhm"
server_key: "50faff955370d7ec8fd17a6638d7862a"
server_key_iv: "30a4836c65a6262e631f342d7a1896a4"
server_key_version: "encryptionkey@0"
I20260810 04:49:54.921869 17469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "cb5873f02f0149cdb8bb9be2686e84cb"
format_stamp: "Formatted at 2026-08-10 04:49:54 on dist-test-slave-8hhm"
server_key: "50faff955370d7ec8fd17a6638d7862a"
server_key_iv: "30a4836c65a6262e631f342d7a1896a4"
server_key_version: "encryptionkey@0"
I20260810 04:49:54.923280 17469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:49:54.924026 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:54.924219 17469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:54.924348 17469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "cb5873f02f0149cdb8bb9be2686e84cb"
format_stamp: "Formatted at 2026-08-10 04:49:54 on dist-test-slave-8hhm"
server_key: "50faff955370d7ec8fd17a6638d7862a"
server_key_iv: "30a4836c65a6262e631f342d7a1896a4"
server_key_version: "encryptionkey@0"
I20260810 04:49:54.924429 17469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:54.934300 17469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:54.936206 17469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:54.936381 17469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:54.941004 17469 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35117
I20260810 04:49:54.941039 17588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35117 every 8 connection(s)
I20260810 04:49:54.941476 17469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:54.942418 17589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:54.944854 17589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb: Bootstrap starting.
I20260810 04:49:54.945130 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 17469
I20260810 04:49:54.945286 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:54.945537 15284 external_mini_cluster.cc:1428] Setting key 7ad0d5bf795afdc6a5fb504c12fdac00
I20260810 04:49:54.945591 17589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:54.945969 17589 log.cc:824] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:54.946810 17589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb: No bootstrap required, opened a new log
I20260810 04:49:54.948349 17589 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5873f02f0149cdb8bb9be2686e84cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35117 } }
I20260810 04:49:54.948539 17589 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:54.948593 17589 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5873f02f0149cdb8bb9be2686e84cb, State: Initialized, Role: FOLLOWER
I20260810 04:49:54.948789 17589 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5873f02f0149cdb8bb9be2686e84cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35117 } }
I20260810 04:49:54.948899 17589 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:54.948948 17589 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:54.949004 17589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:54.949676 17589 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5873f02f0149cdb8bb9be2686e84cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35117 } }
I20260810 04:49:54.949810 17589 leader_election.cc:302] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5873f02f0149cdb8bb9be2686e84cb; no voters:
I20260810 04:49:54.949985 17589 leader_election.cc:288] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:54.950011 17598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:54.950261 17589 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:54.950289 17598 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [term 1 LEADER]: Becoming Leader. State: Replica: cb5873f02f0149cdb8bb9be2686e84cb, State: Running, Role: LEADER
I20260810 04:49:54.950511 17598 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5873f02f0149cdb8bb9be2686e84cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35117 } }
I20260810 04:49:54.951154 17600 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5873f02f0149cdb8bb9be2686e84cb. Latest consensus state: current_term: 1 leader_uuid: "cb5873f02f0149cdb8bb9be2686e84cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5873f02f0149cdb8bb9be2686e84cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35117 } } }
I20260810 04:49:54.951253 17600 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:54.951704 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:54.946237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35841 (local address 127.0.0.1:35117)
0810 04:49:54.946411 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:54.946419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:54.946444 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:49:54.946858 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:54.946864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:54.946906 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:54.947058 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:54.947282 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:54.948133 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:54.949467 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:54.949873 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:54.950003 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:54.950695 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:54.950699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:54.950700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:54.950943 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:54.950984 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:54.950999 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:54.951084 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:54.951407 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 04:49:54.951441 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260810 04:49:54.951856 17599 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5873f02f0149cdb8bb9be2686e84cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5873f02f0149cdb8bb9be2686e84cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35117 } } }
I20260810 04:49:54.952000 17610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:54.952251 17599 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:54.952755 17610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:54.952793 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:49:54.954631 17610 catalog_manager.cc:1347] Generated new cluster ID: b054e4061be144a1991d18323ed1afa5
I20260810 04:49:54.954691 17610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:54.958256 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:54.954594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46545 (local address 127.0.0.1:35117)
0810 04:49:54.954631 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:54.954633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:54.954646 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:49:54.954829 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:54.954830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:54.954841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:54.954904 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:54.955062 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:54.955769 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:54.956834 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:54.957160 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:54.957256 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:54.957999 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:54.958001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:54.958001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:54.958006 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:54.958022 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:54.958026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:54.958092 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:54.958176 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 04:49:54.958186 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:49:54.972263 17610 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:54.973127 17610 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:54.981021 17610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5873f02f0149cdb8bb9be2686e84cb: Generated new TSK 0
I20260810 04:49:54.981244 17610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:49:54.989713 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 17469
W20260810 04:49:55.005589 17435 connection.cc:535] client connection to 127.0.0.1:35117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35117): Connection reset by peer (error 104)
2026-08-10T04:49:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1540 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:55 dist-test-slave-8hhm krb5kdc[17637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:55 dist-test-slave-8hhm krb5kdc[17637](info): set up 2 sockets
Aug 10 04:49:55 dist-test-slave-8hhm krb5kdc[17637](info): commencing operation
krb5kdc: starting...
W20260810 04:49:56.260068 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:49:56 dist-test-slave-8hhm krb5kdc[17637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:49:56.379892 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:56.462409 17660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:56.462608 17660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:56.462630 17660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:56.463698 17660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:56.463732 17660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:56.463745 17660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:56.463758 17660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:56.463770 17660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:56.465257 17660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:56.465436 17660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:56.465605 17660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:56.470480 17670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:56.470459 17674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:56.470492 17660 server_base.cc:1034] running on GCE node
W20260810 04:49:56.470463 17671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:56.471001 17660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:56.471349 17660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:56.472523 17660 hybrid_clock.cc:648] HybridClock initialized: now 1786337396472507 us; error 27 us; skew 500 ppm
Aug 10 04:49:56 dist-test-slave-8hhm krb5kdc[17637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:56.475073 17660 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:49:56.475525 17660 webserver.cc:460] Webserver started at http://127.0.0.1:45279/ using document root <none> and password file <none>
I20260810 04:49:56.475741 17660 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:56.475811 17660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:56.475961 17660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:56.476959 17660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "8896537ee73046c1aaf8d2359b422af4"
format_stamp: "Formatted at 2026-08-10 04:49:56 on dist-test-slave-8hhm"
server_key: "b2ccf0155e416a180d41a5afe9db45f0"
server_key_iv: "88e7fb1bd762f39010b4d8db122dc089"
server_key_version: "encryptionkey@0"
I20260810 04:49:56.477360 17660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "8896537ee73046c1aaf8d2359b422af4"
format_stamp: "Formatted at 2026-08-10 04:49:56 on dist-test-slave-8hhm"
server_key: "b2ccf0155e416a180d41a5afe9db45f0"
server_key_iv: "88e7fb1bd762f39010b4d8db122dc089"
server_key_version: "encryptionkey@0"
I20260810 04:49:56.478727 17660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:49:56.479463 17682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:56.479627 17660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:49:56.479734 17660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "8896537ee73046c1aaf8d2359b422af4"
format_stamp: "Formatted at 2026-08-10 04:49:56 on dist-test-slave-8hhm"
server_key: "b2ccf0155e416a180d41a5afe9db45f0"
server_key_iv: "88e7fb1bd762f39010b4d8db122dc089"
server_key_version: "encryptionkey@0"
I20260810 04:49:56.479806 17660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:56.482999 17660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:56.484843 17660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:56.484989 17660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:56.488840 17660 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41021
I20260810 04:49:56.488869 17770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41021 every 8 connection(s)
I20260810 04:49:56.489169 17660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:56.490110 17771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:56.492159 17771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4: Bootstrap starting.
I20260810 04:49:56.492668 17771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:56.492905 17771 log.cc:824] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:56.493841 17771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4: No bootstrap required, opened a new log
I20260810 04:49:56.494877 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 17660
I20260810 04:49:56.495000 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:56.495231 15284 external_mini_cluster.cc:1428] Setting key 98e6da3f746b4032276b8f85c3f16fda
I20260810 04:49:56.495711 17771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8896537ee73046c1aaf8d2359b422af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41021 } }
I20260810 04:49:56.495936 17771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:56.496016 17771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8896537ee73046c1aaf8d2359b422af4, State: Initialized, Role: FOLLOWER
I20260810 04:49:56.496150 17771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8896537ee73046c1aaf8d2359b422af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41021 } }
I20260810 04:49:56.496237 17771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:56.496292 17771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:56.496343 17771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:56.497354 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:56.495752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37393 (local address 127.0.0.1:41021)
0810 04:49:56.495937 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:56.495946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:56.495976 (+ 30us) server_negotiation.cc:407] Connection header received
0810 04:49:56.496105 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:56.496112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:56.496141 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:56.496293 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:56.496912 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:56.496915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:56.496916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:56.496919 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:56.496966 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:56.496977 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:56.497045 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:56.497112 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 04:49:56.497152 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260810 04:49:56.497440 17771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8896537ee73046c1aaf8d2359b422af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41021 } }
I20260810 04:49:56.497558 17771 leader_election.cc:302] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8896537ee73046c1aaf8d2359b422af4; no voters:
I20260810 04:49:56.497712 17771 leader_election.cc:288] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:56.497749 17779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:56.497913 17779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [term 1 LEADER]: Becoming Leader. State: Replica: 8896537ee73046c1aaf8d2359b422af4, State: Running, Role: LEADER
I20260810 04:49:56.497958 17771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:56.498044 17779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8896537ee73046c1aaf8d2359b422af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41021 } }
I20260810 04:49:56.498325 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:49:56.499747 17781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8896537ee73046c1aaf8d2359b422af4. Latest consensus state: current_term: 1 leader_uuid: "8896537ee73046c1aaf8d2359b422af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8896537ee73046c1aaf8d2359b422af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41021 } } }
I20260810 04:49:56.499871 17781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:56.500105 17785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8896537ee73046c1aaf8d2359b422af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8896537ee73046c1aaf8d2359b422af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41021 } } }
I20260810 04:49:56.500216 17785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4 [sys.catalog]: This master's current role is: LEADER
W20260810 04:49:56.500372 17806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:49:56.500422 17806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:49:56.500480 17818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:56.500618 17818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:56.501130 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:56.500034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34393 (local address 127.0.0.1:41021)
0810 04:49:56.500068 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:56.500070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:56.500141 (+ 71us) server_negotiation.cc:407] Connection header received
0810 04:49:56.500207 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:56.500208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:56.500223 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:56.500277 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:56.500911 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:56.500912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:56.500912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:56.500916 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:56.500929 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:56.500933 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:56.500983 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:56.501032 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 04:49:56.501047 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:49:56.502171 17818 catalog_manager.cc:1347] Generated new cluster ID: 998442f87d62419bab96454517035553
I20260810 04:49:56.502228 17818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:56.511272 17818 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:56.511418 17818 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:56.516541 17818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8896537ee73046c1aaf8d2359b422af4: Generated new TSK 0
I20260810 04:49:56.516703 17818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:49:56.520133 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 17660
2026-08-10T04:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1531 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:56 dist-test-slave-8hhm krb5kdc[17825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:56 dist-test-slave-8hhm krb5kdc[17825](info): set up 2 sockets
Aug 10 04:49:56 dist-test-slave-8hhm krb5kdc[17825](info): commencing operation
krb5kdc: starting...
W20260810 04:49:57.771210 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:49:57 dist-test-slave-8hhm krb5kdc[17825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:49:57.889153 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:57.969262 17852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:57.969516 17852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:57.969556 17852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:57.970697 17852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:57.970798 17852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:57.970824 17852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:57.970851 17852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:57.970896 17852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:57.972383 17852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:57.972661 17852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:57.972892 17852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:57.977963 17858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:57.977970 17859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:57.977977 17861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:57.978334 17852 server_base.cc:1034] running on GCE node
I20260810 04:49:57.978614 17852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:57.978925 17852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:57.980094 17852 hybrid_clock.cc:648] HybridClock initialized: now 1786337397980070 us; error 30 us; skew 500 ppm
Aug 10 04:49:57 dist-test-slave-8hhm krb5kdc[17825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:57.982688 17852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:49:57.983096 17852 webserver.cc:460] Webserver started at http://127.0.0.1:43509/ using document root <none> and password file <none>
I20260810 04:49:57.983340 17852 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:57.983426 17852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:57.983573 17852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:57.984611 17852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "f103cfb2471d4d35883ce01ae84ad965"
format_stamp: "Formatted at 2026-08-10 04:49:57 on dist-test-slave-8hhm"
server_key: "4d9e2bc18f0d082222e15bca1c9de760"
server_key_iv: "6ec77e0a91d880efbdc1e818513e66ac"
server_key_version: "encryptionkey@0"
I20260810 04:49:57.984970 17852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "f103cfb2471d4d35883ce01ae84ad965"
format_stamp: "Formatted at 2026-08-10 04:49:57 on dist-test-slave-8hhm"
server_key: "4d9e2bc18f0d082222e15bca1c9de760"
server_key_iv: "6ec77e0a91d880efbdc1e818513e66ac"
server_key_version: "encryptionkey@0"
I20260810 04:49:57.986313 17852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:49:57.987012 17877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:57.987191 17852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:57.987289 17852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "f103cfb2471d4d35883ce01ae84ad965"
format_stamp: "Formatted at 2026-08-10 04:49:57 on dist-test-slave-8hhm"
server_key: "4d9e2bc18f0d082222e15bca1c9de760"
server_key_iv: "6ec77e0a91d880efbdc1e818513e66ac"
server_key_version: "encryptionkey@0"
I20260810 04:49:57.987370 17852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:58.007921 17852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:58.009768 17852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:58.009927 17852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:58.013844 17852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37031
I20260810 04:49:58.013890 17969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37031 every 8 connection(s)
I20260810 04:49:58.014257 17852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:58.015115 17970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:58.017438 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965: Bootstrap starting.
I20260810 04:49:58.018018 17970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:58.018308 17970 log.cc:824] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:58.018989 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965: No bootstrap required, opened a new log
I20260810 04:49:58.020507 17970 raft_consensus.cc:348] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f103cfb2471d4d35883ce01ae84ad965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37031 } }
I20260810 04:49:58.020641 17970 raft_consensus.cc:374] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:58.020663 17970 raft_consensus.cc:729] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f103cfb2471d4d35883ce01ae84ad965, State: Initialized, Role: FOLLOWER
I20260810 04:49:58.020828 17970 consensus_queue.cc:260] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f103cfb2471d4d35883ce01ae84ad965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37031 } }
I20260810 04:49:58.020891 17970 raft_consensus.cc:388] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:58.020911 17970 raft_consensus.cc:482] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:58.020980 17970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:58.021512 17970 raft_consensus.cc:504] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f103cfb2471d4d35883ce01ae84ad965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37031 } }
I20260810 04:49:58.021622 17970 leader_election.cc:302] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f103cfb2471d4d35883ce01ae84ad965; no voters:
I20260810 04:49:58.021812 17970 leader_election.cc:288] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:58.021878 17975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:58.022048 17975 raft_consensus.cc:686] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [term 1 LEADER]: Becoming Leader. State: Replica: f103cfb2471d4d35883ce01ae84ad965, State: Running, Role: LEADER
I20260810 04:49:58.022059 17970 sys_catalog.cc:564] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:58.022186 17975 consensus_queue.cc:237] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f103cfb2471d4d35883ce01ae84ad965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37031 } }
I20260810 04:49:58.022572 17979 sys_catalog.cc:455] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f103cfb2471d4d35883ce01ae84ad965. Latest consensus state: current_term: 1 leader_uuid: "f103cfb2471d4d35883ce01ae84ad965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f103cfb2471d4d35883ce01ae84ad965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37031 } } }
I20260810 04:49:58.022672 17979 sys_catalog.cc:458] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:58.022724 17977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f103cfb2471d4d35883ce01ae84ad965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f103cfb2471d4d35883ce01ae84ad965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37031 } } }
I20260810 04:49:58.022795 17977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:58.022913 17991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:58.023550 17991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:58.023914 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 17852
I20260810 04:49:58.024061 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:58.024248 15284 external_mini_cluster.cc:1428] Setting key 67b401eba527220808cb71e036b7cd4a
I20260810 04:49:58.025372 17991 catalog_manager.cc:1347] Generated new cluster ID: 3cfc9508f8c24c05a394fcd774152662
I20260810 04:49:58.025425 17991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:58.030946 17991 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:58.031598 17991 catalog_manager.cc:1504] Loading token signing keys...
Aug 10 04:49:58 dist-test-slave-8hhm krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:49:58.034094 17991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f103cfb2471d4d35883ce01ae84ad965: Generated new TSK 0
I20260810 04:49:58.034252 17991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:49:58.036278 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:58.024881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57727 (local address 127.0.0.1:37031)
0810 04:49:58.025058 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:58.025066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:58.025099 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:49:58.025240 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:58.025247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:58.025316 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:58.025451 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:58.026359 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:58.027680 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:58.028746 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:58.029331 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:58.029433 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:58.032393 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:58.032397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:58.032398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:58.032409 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:58.034636 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:58.035043 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:58.035046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:58.035047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:58.035164 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:58.035386 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:58.035387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:58.035387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:58.035701 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:58.035837 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:58.035953 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 04:49:58.036095 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260810 04:49:58.037312 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:49:58.044414 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:58.038700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46537 (local address 127.0.0.1:37031)
0810 04:49:58.038747 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:58.038749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:58.038829 (+ 80us) server_negotiation.cc:407] Connection header received
0810 04:49:58.038897 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:58.038898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:58.038908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:58.038977 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:58.039473 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:58.039945 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:58.040672 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:58.040877 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:58.040953 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:58.042067 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:58.042068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:58.042069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:49:58.042076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:58.043297 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:58.043668 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:58.043670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:58.043670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:58.043729 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:49:58.043931 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:49:58.043932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:49:58.043932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:49:58.044037 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:58.044137 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:58.044243 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 04:49:58.044321 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:49:58.046211 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 17852
2026-08-10T04:49:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1523 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:58 dist-test-slave-8hhm krb5kdc[18030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:58 dist-test-slave-8hhm krb5kdc[18030](info): set up 2 sockets
Aug 10 04:49:58 dist-test-slave-8hhm krb5kdc[18030](info): commencing operation
krb5kdc: starting...
W20260810 04:49:59.302698 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:49:59 dist-test-slave-8hhm krb5kdc[18030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:49:59.420518 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:49:59.501665 18058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:49:59.501853 18058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:49:59.501875 18058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:49:59.502954 18058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:49:59.502988 18058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:49:59.503002 18058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:49:59.503014 18058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:49:59.503026 18058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:49:59.504388 18058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:49:59.504563 18058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:49:59.504732 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:49:59.509392 18065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:59.509352 18072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:49:59.509361 18064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:49:59.509649 18058 server_base.cc:1034] running on GCE node
I20260810 04:49:59.509877 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:49:59.510097 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:49:59.511281 18058 hybrid_clock.cc:648] HybridClock initialized: now 1786337399511270 us; error 30 us; skew 500 ppm
Aug 10 04:49:59 dist-test-slave-8hhm krb5kdc[18030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:49:59.516212 18058 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:49:59.516621 18058 webserver.cc:460] Webserver started at http://127.0.0.1:44307/ using document root <none> and password file <none>
I20260810 04:49:59.516837 18058 fs_manager.cc:359] Metadata directory not provided
I20260810 04:49:59.516912 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:49:59.517063 18058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:49:59.518132 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "72259200c1e940e791418612b3d983a5"
format_stamp: "Formatted at 2026-08-10 04:49:59 on dist-test-slave-8hhm"
server_key: "fcae6f026a061a89221bf14877fd3c95"
server_key_iv: "c83977f2af590741a46656cb7131992b"
server_key_version: "encryptionkey@0"
I20260810 04:49:59.518477 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "72259200c1e940e791418612b3d983a5"
format_stamp: "Formatted at 2026-08-10 04:49:59 on dist-test-slave-8hhm"
server_key: "fcae6f026a061a89221bf14877fd3c95"
server_key_iv: "c83977f2af590741a46656cb7131992b"
server_key_version: "encryptionkey@0"
I20260810 04:49:59.519729 18058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:49:59.520401 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:49:59.520591 18058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:49:59.520704 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "72259200c1e940e791418612b3d983a5"
format_stamp: "Formatted at 2026-08-10 04:49:59 on dist-test-slave-8hhm"
server_key: "fcae6f026a061a89221bf14877fd3c95"
server_key_iv: "c83977f2af590741a46656cb7131992b"
server_key_version: "encryptionkey@0"
I20260810 04:49:59.520792 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:49:59.529191 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:49:59.530957 18058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:49:59.531108 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:49:59.535022 18058 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40779
I20260810 04:49:59.535054 18182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40779 every 8 connection(s)
I20260810 04:49:59.535374 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:49:59.536243 18184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:49:59.538529 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5: Bootstrap starting.
I20260810 04:49:59.539108 18184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5: Neither blocks nor log segments found. Creating new log.
I20260810 04:49:59.539414 18184 log.cc:824] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5: Log is configured to *not* fsync() on all Append() calls
I20260810 04:49:59.540122 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5: No bootstrap required, opened a new log
I20260810 04:49:59.541715 18184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72259200c1e940e791418612b3d983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40779 } }
I20260810 04:49:59.541854 18184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:49:59.541904 18184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72259200c1e940e791418612b3d983a5, State: Initialized, Role: FOLLOWER
I20260810 04:49:59.542052 18184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72259200c1e940e791418612b3d983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40779 } }
I20260810 04:49:59.542135 18184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:49:59.542178 18184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:49:59.542228 18184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:49:59.542704 18184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72259200c1e940e791418612b3d983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40779 } }
I20260810 04:49:59.542824 18184 leader_election.cc:302] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72259200c1e940e791418612b3d983a5; no voters:
I20260810 04:49:59.543002 18184 leader_election.cc:288] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:49:59.543044 18188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:49:59.543239 18188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [term 1 LEADER]: Becoming Leader. State: Replica: 72259200c1e940e791418612b3d983a5, State: Running, Role: LEADER
I20260810 04:49:59.543249 18184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:49:59.543344 18188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72259200c1e940e791418612b3d983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40779 } }
I20260810 04:49:59.543709 18190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72259200c1e940e791418612b3d983a5. Latest consensus state: current_term: 1 leader_uuid: "72259200c1e940e791418612b3d983a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72259200c1e940e791418612b3d983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40779 } } }
I20260810 04:49:59.543706 18189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72259200c1e940e791418612b3d983a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72259200c1e940e791418612b3d983a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40779 } } }
I20260810 04:49:59.543831 18190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:59.543839 18189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5 [sys.catalog]: This master's current role is: LEADER
I20260810 04:49:59.544034 18194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:49:59.544592 18194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:49:59.544979 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18058
I20260810 04:49:59.545091 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:49:59.545331 15284 external_mini_cluster.cc:1428] Setting key d6844528402c30a30831db625dd716bf
I20260810 04:49:59.546917 18194 catalog_manager.cc:1347] Generated new cluster ID: 68ab9e3d01b745de9d6c4e250f514c96
I20260810 04:49:59.547004 18194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:49:59.550760 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:59.545844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45247 (local address 127.0.0.1:40779)
0810 04:49:59.546358 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:59.546367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:59.546397 (+ 30us) server_negotiation.cc:407] Connection header received
0810 04:49:59.546506 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:59.546512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:59.546565 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:59.546707 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:59.546963 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:59.547901 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:59.548743 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:59.549192 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:59.549341 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:59.550090 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:59.550094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:59.550094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:59.550278 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:59.550310 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:59.550320 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:59.550407 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:59.550481 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 04:49:59.550545 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":78,"threads_started":1}
I20260810 04:49:59.551678 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:49:59.556393 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:49:59.553583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45123 (local address 127.0.0.1:40779)
0810 04:49:59.553628 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:49:59.553630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:49:59.553818 (+ 188us) server_negotiation.cc:407] Connection header received
0810 04:49:59.553859 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:49:59.553860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:49:59.553870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:49:59.553916 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:49:59.554020 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:59.554505 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:59.555207 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:49:59.555408 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:49:59.555487 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:49:59.556163 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:49:59.556164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:49:59.556164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:49:59.556169 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:49:59.556183 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:49:59.556187 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:49:59.556234 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:49:59.556290 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 04:49:59.556312 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:49:59.565141 18194 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:49:59.565683 18194 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:49:59.569900 18194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72259200c1e940e791418612b3d983a5: Generated new TSK 0
I20260810 04:49:59.570079 18194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:49:59.573489 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18058
2026-08-10T04:49:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1527 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:49:59 dist-test-slave-8hhm krb5kdc[18234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:49:59 dist-test-slave-8hhm krb5kdc[18234](info): set up 2 sockets
Aug 10 04:49:59 dist-test-slave-8hhm krb5kdc[18234](info): commencing operation
krb5kdc: starting...
W20260810 04:50:00.832028 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:00 dist-test-slave-8hhm krb5kdc[18234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:50:00.953585 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:01.035228 18262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:01.035422 18262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:01.035444 18262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:01.036579 18262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:01.036625 18262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:01.036639 18262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:01.036651 18262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:01.036664 18262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:01.038201 18262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:01.038435 18262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:01.038672 18262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:01.043464 18275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:01.043457 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:01.043522 18274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:01.043973 18262 server_base.cc:1034] running on GCE node
I20260810 04:50:01.044242 18262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:01.044528 18262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:01.045715 18262 hybrid_clock.cc:648] HybridClock initialized: now 1786337401045689 us; error 30 us; skew 500 ppm
Aug 10 04:50:01 dist-test-slave-8hhm krb5kdc[18234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:01.048228 18262 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:50:01.048605 18262 webserver.cc:460] Webserver started at http://127.0.0.1:36011/ using document root <none> and password file <none>
I20260810 04:50:01.048779 18262 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:01.048818 18262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:01.048905 18262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:01.049913 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "8be4fe5af9ee4b3f915123e687e07e81"
format_stamp: "Formatted at 2026-08-10 04:50:01 on dist-test-slave-8hhm"
server_key: "4fb3e7cf3a3e0c1b8dbe19a18dc6b3db"
server_key_iv: "e6b6e3b16c1b9714970fc6dfcfa88090"
server_key_version: "encryptionkey@0"
I20260810 04:50:01.050223 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "8be4fe5af9ee4b3f915123e687e07e81"
format_stamp: "Formatted at 2026-08-10 04:50:01 on dist-test-slave-8hhm"
server_key: "4fb3e7cf3a3e0c1b8dbe19a18dc6b3db"
server_key_iv: "e6b6e3b16c1b9714970fc6dfcfa88090"
server_key_version: "encryptionkey@0"
I20260810 04:50:01.051468 18262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:50:01.052233 18287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:01.052423 18262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:50:01.052556 18262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "8be4fe5af9ee4b3f915123e687e07e81"
format_stamp: "Formatted at 2026-08-10 04:50:01 on dist-test-slave-8hhm"
server_key: "4fb3e7cf3a3e0c1b8dbe19a18dc6b3db"
server_key_iv: "e6b6e3b16c1b9714970fc6dfcfa88090"
server_key_version: "encryptionkey@0"
I20260810 04:50:01.052646 18262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:01.055979 18262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:01.057937 18262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:01.058094 18262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:01.061913 18262 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46845
I20260810 04:50:01.061941 18376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46845 every 8 connection(s)
I20260810 04:50:01.062309 18262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:01.063310 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:01.065641 18379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81: Bootstrap starting.
I20260810 04:50:01.066227 18379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:01.066517 18379 log.cc:824] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:01.067211 18379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81: No bootstrap required, opened a new log
I20260810 04:50:01.068306 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18262
I20260810 04:50:01.068435 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:01.068670 15284 external_mini_cluster.cc:1428] Setting key 6599cde510142631a794338ba7ec99f1
I20260810 04:50:01.069362 18379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be4fe5af9ee4b3f915123e687e07e81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } }
I20260810 04:50:01.069530 18379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:01.069574 18379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be4fe5af9ee4b3f915123e687e07e81, State: Initialized, Role: FOLLOWER
I20260810 04:50:01.069737 18379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be4fe5af9ee4b3f915123e687e07e81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } }
I20260810 04:50:01.069834 18379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:01.069871 18379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:01.070025 18379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:01.070796 18379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be4fe5af9ee4b3f915123e687e07e81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } }
I20260810 04:50:01.070917 18379 leader_election.cc:302] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8be4fe5af9ee4b3f915123e687e07e81; no voters:
I20260810 04:50:01.071085 18379 leader_election.cc:288] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:01.071107 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:01.069205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48015 (local address 127.0.0.1:46845)
0810 04:50:01.069441 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:01.069448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:01.069472 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:50:01.069583 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:01.069590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:01.069636 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:01.069781 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:01.070639 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:01.070643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:01.070643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:01.070646 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:01.070678 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:01.070689 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:01.070744 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:01.070840 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 04:50:01.070890 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260810 04:50:01.071132 18388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:01.071341 18388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [term 1 LEADER]: Becoming Leader. State: Replica: 8be4fe5af9ee4b3f915123e687e07e81, State: Running, Role: LEADER
I20260810 04:50:01.071341 18379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:01.071481 18388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be4fe5af9ee4b3f915123e687e07e81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } }
I20260810 04:50:01.071856 18390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8be4fe5af9ee4b3f915123e687e07e81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be4fe5af9ee4b3f915123e687e07e81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } } }
I20260810 04:50:01.071862 18391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8be4fe5af9ee4b3f915123e687e07e81. Latest consensus state: current_term: 1 leader_uuid: "8be4fe5af9ee4b3f915123e687e07e81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be4fe5af9ee4b3f915123e687e07e81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } } }
I20260810 04:50:01.071969 18390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:01.072036 18391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:01.072460 18402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:01.072839 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:01.073122 18402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:01.074838 18402 catalog_manager.cc:1347] Generated new cluster ID: ed6765c2d3bd40478bdcc79cda5a1fc7
I20260810 04:50:01.074908 18402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:01.078740 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:01.077519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47139 (local address 127.0.0.1:46845)
0810 04:50:01.077563 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:01.077565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:01.077747 (+ 182us) server_negotiation.cc:407] Connection header received
0810 04:50:01.077850 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:01.077851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:01.077856 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:01.077912 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:01.078510 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:01.078511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:01.078511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:01.078515 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:01.078529 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:01.078534 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:01.078587 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:01.078655 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 04:50:01.078667 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:50:01.089129 18402 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:01.089330 18402 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:01.092921 18402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8be4fe5af9ee4b3f915123e687e07e81: Generated new TSK 0
I20260810 04:50:01.093113 18402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:01.097759 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18262
2026-08-10T04:50:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1536 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:01 dist-test-slave-8hhm krb5kdc[18428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:01 dist-test-slave-8hhm krb5kdc[18428](info): set up 2 sockets
Aug 10 04:50:01 dist-test-slave-8hhm krb5kdc[18428](info): commencing operation
krb5kdc: starting...
W20260810 04:50:02.374691 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:02 dist-test-slave-8hhm krb5kdc[18428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:50:02.497586 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:02.580243 18457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:02.580443 18457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:02.580466 18457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:02.581674 18457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:02.581724 18457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:02.581738 18457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:02.581781 18457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:02.581809 18457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:02.583328 18457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:02.583544 18457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:02.583777 18457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:02.588698 18468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:02.588711 18473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:02.588685 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:02.588907 18457 server_base.cc:1034] running on GCE node
I20260810 04:50:02.589167 18457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:02.589515 18457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:02.590664 18457 hybrid_clock.cc:648] HybridClock initialized: now 1786337402590646 us; error 31 us; skew 500 ppm
Aug 10 04:50:02 dist-test-slave-8hhm krb5kdc[18428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:02.593122 18457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:50:02.593580 18457 webserver.cc:460] Webserver started at http://127.0.0.1:42439/ using document root <none> and password file <none>
I20260810 04:50:02.593817 18457 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:02.593887 18457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:02.593986 18457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:02.595065 18457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "25ad8604541a4dc39b77633a52076267"
format_stamp: "Formatted at 2026-08-10 04:50:02 on dist-test-slave-8hhm"
server_key: "57efa27b94d99e43f460a6e519a0949b"
server_key_iv: "a4bc00bfe02682f9ab1a2e5bd97ff159"
server_key_version: "encryptionkey@0"
I20260810 04:50:02.595386 18457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "25ad8604541a4dc39b77633a52076267"
format_stamp: "Formatted at 2026-08-10 04:50:02 on dist-test-slave-8hhm"
server_key: "57efa27b94d99e43f460a6e519a0949b"
server_key_iv: "a4bc00bfe02682f9ab1a2e5bd97ff159"
server_key_version: "encryptionkey@0"
I20260810 04:50:02.596668 18457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:50:02.597395 18482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:02.597573 18457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:50:02.597680 18457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "25ad8604541a4dc39b77633a52076267"
format_stamp: "Formatted at 2026-08-10 04:50:02 on dist-test-slave-8hhm"
server_key: "57efa27b94d99e43f460a6e519a0949b"
server_key_iv: "a4bc00bfe02682f9ab1a2e5bd97ff159"
server_key_version: "encryptionkey@0"
I20260810 04:50:02.597765 18457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:02.620374 18457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:02.622412 18457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:02.622584 18457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:02.626703 18457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38573
I20260810 04:50:02.626739 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38573 every 8 connection(s)
I20260810 04:50:02.627126 18457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:02.628039 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:02.630424 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267: Bootstrap starting.
I20260810 04:50:02.630970 18577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:02.631199 18577 log.cc:824] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:02.631896 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267: No bootstrap required, opened a new log
I20260810 04:50:02.632838 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18457
I20260810 04:50:02.632962 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:02.633179 15284 external_mini_cluster.cc:1428] Setting key 7dc58851bef3b469de4a8ccf338abeb1
I20260810 04:50:02.634122 18577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ad8604541a4dc39b77633a52076267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } }
I20260810 04:50:02.634266 18577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:02.634289 18577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ad8604541a4dc39b77633a52076267, State: Initialized, Role: FOLLOWER
I20260810 04:50:02.634440 18577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ad8604541a4dc39b77633a52076267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } }
I20260810 04:50:02.634536 18577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:02.634569 18577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:02.634624 18577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:02.635687 18577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ad8604541a4dc39b77633a52076267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } }
I20260810 04:50:02.635787 18577 leader_election.cc:302] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ad8604541a4dc39b77633a52076267; no voters:
I20260810 04:50:02.636006 18577 leader_election.cc:288] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:02.636029 18583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:02.636143 18583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [term 1 LEADER]: Becoming Leader. State: Replica: 25ad8604541a4dc39b77633a52076267, State: Running, Role: LEADER
I20260810 04:50:02.636277 18577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:02.636300 18583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ad8604541a4dc39b77633a52076267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } }
I20260810 04:50:02.636762 18584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ad8604541a4dc39b77633a52076267. Latest consensus state: current_term: 1 leader_uuid: "25ad8604541a4dc39b77633a52076267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ad8604541a4dc39b77633a52076267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } } }
I20260810 04:50:02.636869 18584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:02.637084 18587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ad8604541a4dc39b77633a52076267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ad8604541a4dc39b77633a52076267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38573 } } }
I20260810 04:50:02.637319 18587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:02.637501 18601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:02.638023 18601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:02.640347 18601 catalog_manager.cc:1347] Generated new cluster ID: 97e4d8acd6f74d5f84c8799aa68cf1dc
I20260810 04:50:02.640403 18601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:50:02 dist-test-slave-8hhm krb5kdc[18428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:50:02.644496 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:02.633834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:40005 (local address 127.0.0.1:38573)
0810 04:50:02.634033 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:02.634041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:02.634067 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:50:02.634164 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:02.634170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:02.634213 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:02.634387 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:02.635272 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:02.636411 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:02.637472 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:02.638153 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:02.638260 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:02.641004 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:02.641008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:02.641009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:02.641021 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:02.642826 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:02.643231 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:02.643234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:02.643235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:02.643349 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:02.643584 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:02.643585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:02.643585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:02.643853 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:02.644036 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:02.644122 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:50:02.644345 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260810 04:50:02.645500 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:02.664448 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:02.657524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:39063 (local address 127.0.0.1:38573)
0810 04:50:02.657582 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:02.657584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:02.657733 (+ 149us) server_negotiation.cc:407] Connection header received
0810 04:50:02.657835 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:02.657836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:02.657845 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:02.657898 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:02.658433 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:02.658905 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:02.659567 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:02.659784 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:02.659878 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:02.661119 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:02.661121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:02.661121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:02.661129 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:02.663181 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:02.663573 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:02.663574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:02.663574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:02.663639 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:02.663888 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:02.663889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:02.663890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:02.664008 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:02.664122 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:02.664256 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 04:50:02.664323 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:50:02.667841 18601 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:02.668349 18601 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:02.676754 18601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25ad8604541a4dc39b77633a52076267: Generated new TSK 0
I20260810 04:50:02.676950 18601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:02.682543 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18457
W20260810 04:50:02.690415 18426 connection.cc:535] client connection to 127.0.0.1:38573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38573): Connection reset by peer (error 104)
2026-08-10T04:50:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1565 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:02 dist-test-slave-8hhm krb5kdc[18633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:02 dist-test-slave-8hhm krb5kdc[18633](info): set up 2 sockets
Aug 10 04:50:02 dist-test-slave-8hhm krb5kdc[18633](info): commencing operation
krb5kdc: starting...
W20260810 04:50:03.929183 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:04 dist-test-slave-8hhm krb5kdc[18633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:50:04.051651 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:04.134445 18654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:04.134655 18654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:04.134678 18654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:04.135738 18654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:04.135771 18654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:04.135784 18654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:04.135797 18654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:04.135809 18654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:04.137190 18654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:04.137449 18654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:04.137616 18654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:04.142395 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:04.142408 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:04.142436 18654 server_base.cc:1034] running on GCE node
W20260810 04:50:04.142441 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:04.142884 18654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:04.143165 18654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:04.144309 18654 hybrid_clock.cc:648] HybridClock initialized: now 1786337404144289 us; error 37 us; skew 500 ppm
Aug 10 04:50:04 dist-test-slave-8hhm krb5kdc[18633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:04.146750 18654 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:50:04.147156 18654 webserver.cc:460] Webserver started at http://127.0.0.1:36961/ using document root <none> and password file <none>
I20260810 04:50:04.147377 18654 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:04.147444 18654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:04.147598 18654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:04.148634 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "f0b8e1ff5e764223b53193c6908ea4df"
format_stamp: "Formatted at 2026-08-10 04:50:04 on dist-test-slave-8hhm"
server_key: "ef7cc699e23d2f14521aa5b71df9c35e"
server_key_iv: "d7feae31f6dea8f158896c135b81f393"
server_key_version: "encryptionkey@0"
I20260810 04:50:04.148999 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "f0b8e1ff5e764223b53193c6908ea4df"
format_stamp: "Formatted at 2026-08-10 04:50:04 on dist-test-slave-8hhm"
server_key: "ef7cc699e23d2f14521aa5b71df9c35e"
server_key_iv: "d7feae31f6dea8f158896c135b81f393"
server_key_version: "encryptionkey@0"
I20260810 04:50:04.150490 18654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:50:04.151316 18678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:04.151532 18654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:50:04.151638 18654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "f0b8e1ff5e764223b53193c6908ea4df"
format_stamp: "Formatted at 2026-08-10 04:50:04 on dist-test-slave-8hhm"
server_key: "ef7cc699e23d2f14521aa5b71df9c35e"
server_key_iv: "d7feae31f6dea8f158896c135b81f393"
server_key_version: "encryptionkey@0"
I20260810 04:50:04.151732 18654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:04.173671 18654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:04.175705 18654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:04.175870 18654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:04.179996 18654 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45763
I20260810 04:50:04.180037 18757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45763 every 8 connection(s)
I20260810 04:50:04.180416 18654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:04.181473 18758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:04.183938 18758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df: Bootstrap starting.
I20260810 04:50:04.184568 18758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:04.184870 18758 log.cc:824] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:04.185727 18758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df: No bootstrap required, opened a new log
I20260810 04:50:04.186950 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18654
I20260810 04:50:04.187076 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:04.187282 15284 external_mini_cluster.cc:1428] Setting key c556ecb3c817053e78308f9d37d3e974
I20260810 04:50:04.187599 18758 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b8e1ff5e764223b53193c6908ea4df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } }
I20260810 04:50:04.187808 18758 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:04.187860 18758 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0b8e1ff5e764223b53193c6908ea4df, State: Initialized, Role: FOLLOWER
I20260810 04:50:04.188026 18758 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b8e1ff5e764223b53193c6908ea4df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } }
I20260810 04:50:04.188163 18758 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:04.188215 18758 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:04.188271 18758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:04.188863 18758 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b8e1ff5e764223b53193c6908ea4df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } }
I20260810 04:50:04.188992 18758 leader_election.cc:302] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0b8e1ff5e764223b53193c6908ea4df; no voters:
I20260810 04:50:04.189195 18758 leader_election.cc:288] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:04.189275 18763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:04.189407 18763 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [term 1 LEADER]: Becoming Leader. State: Replica: f0b8e1ff5e764223b53193c6908ea4df, State: Running, Role: LEADER
I20260810 04:50:04.189492 18758 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:04.189620 18763 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b8e1ff5e764223b53193c6908ea4df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } }
I20260810 04:50:04.190014 18763 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0b8e1ff5e764223b53193c6908ea4df. Latest consensus state: current_term: 1 leader_uuid: "f0b8e1ff5e764223b53193c6908ea4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b8e1ff5e764223b53193c6908ea4df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } } }
I20260810 04:50:04.190112 18765 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0b8e1ff5e764223b53193c6908ea4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b8e1ff5e764223b53193c6908ea4df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45763 } } }
I20260810 04:50:04.190209 18765 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:04.190166 18763 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:04.190440 18776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:04.191025 18776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:04.192770 18776 catalog_manager.cc:1347] Generated new cluster ID: 8cf71f82f4bc4ca7b1f25dda1daf7baa
I20260810 04:50:04.192826 18776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:04.193933 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:04.187869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:45963 (local address 127.0.0.1:45763)
0810 04:50:04.188072 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:04.188081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:04.188136 (+ 55us) server_negotiation.cc:407] Connection header received
0810 04:50:04.188269 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:04.188276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:04.188335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:04.188472 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:04.188998 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:04.190380 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:04.191623 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:04.192229 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:04.192355 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:04.193162 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:04.193165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:04.193166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:04.193418 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:04.193457 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:04.193471 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:04.193553 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:04.193696 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 04:50:04.193722 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260810 04:50:04.195034 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:04.200384 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:04.196665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:43329 (local address 127.0.0.1:45763)
0810 04:50:04.196697 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:04.196699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:04.196874 (+ 175us) server_negotiation.cc:407] Connection header received
0810 04:50:04.196929 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:04.196929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:04.196939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:04.196987 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:04.197130 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:04.197902 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:04.199001 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:04.199327 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:04.199389 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:04.200106 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:04.200108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:04.200108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:04.200112 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:04.200126 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:04.200130 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:04.200183 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:04.200296 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:50:04.200310 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:50:04.203692 18776 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:04.204185 18776 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:04.211344 18776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0b8e1ff5e764223b53193c6908ea4df: Generated new TSK 0
I20260810 04:50:04.211531 18776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:04.216266 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18654
W20260810 04:50:04.226214 18631 connection.cc:535] client connection to 127.0.0.1:45763 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45763): Connection reset by peer (error 104)
2026-08-10T04:50:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1535 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:04 dist-test-slave-8hhm krb5kdc[18803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:04 dist-test-slave-8hhm krb5kdc[18803](info): set up 2 sockets
Aug 10 04:50:04 dist-test-slave-8hhm krb5kdc[18803](info): commencing operation
krb5kdc: starting...
W20260810 04:50:05.503407 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:05 dist-test-slave-8hhm krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:50:05.627846 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:05.710798 18823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:05.710999 18823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:05.711020 18823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:05.712195 18823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:05.712244 18823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:05.712260 18823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:05.712272 18823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:05.712323 18823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:05.713876 18823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:05.714138 18823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:05.714380 18823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:05.719408 18833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:05.719449 18832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:05.719417 18837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:05.719672 18823 server_base.cc:1034] running on GCE node
I20260810 04:50:05.719926 18823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:05.720244 18823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:05.721405 18823 hybrid_clock.cc:648] HybridClock initialized: now 1786337405721395 us; error 40 us; skew 500 ppm
Aug 10 04:50:05 dist-test-slave-8hhm krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:05.727355 18823 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:50:05.727847 18823 webserver.cc:460] Webserver started at http://127.0.0.1:39265/ using document root <none> and password file <none>
I20260810 04:50:05.728093 18823 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:05.728158 18823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:05.728325 18823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:05.729430 18823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "c472f3814cdd450681e766e6082cf723"
format_stamp: "Formatted at 2026-08-10 04:50:05 on dist-test-slave-8hhm"
server_key: "3558ed64de4d0b6b178586b746a6383a"
server_key_iv: "205a435965e6ddb6d22f6f679ed6937c"
server_key_version: "encryptionkey@0"
I20260810 04:50:05.729794 18823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "c472f3814cdd450681e766e6082cf723"
format_stamp: "Formatted at 2026-08-10 04:50:05 on dist-test-slave-8hhm"
server_key: "3558ed64de4d0b6b178586b746a6383a"
server_key_iv: "205a435965e6ddb6d22f6f679ed6937c"
server_key_version: "encryptionkey@0"
I20260810 04:50:05.731101 18823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:50:05.731885 18851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:05.732062 18823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:50:05.732168 18823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "c472f3814cdd450681e766e6082cf723"
format_stamp: "Formatted at 2026-08-10 04:50:05 on dist-test-slave-8hhm"
server_key: "3558ed64de4d0b6b178586b746a6383a"
server_key_iv: "205a435965e6ddb6d22f6f679ed6937c"
server_key_version: "encryptionkey@0"
I20260810 04:50:05.732252 18823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:05.735683 18823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:05.737569 18823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:05.737715 18823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:05.741650 18823 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45627
I20260810 04:50:05.741678 18940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45627 every 8 connection(s)
I20260810 04:50:05.741989 18823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:05.742982 18941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:05.743008 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 18823
I20260810 04:50:05.743131 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:05.743319 15284 external_mini_cluster.cc:1428] Setting key 1f72c74ef46721413dafac9d6c8c1210
I20260810 04:50:05.745910 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:05.743889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:46519 (local address 127.0.0.1:45627)
0810 04:50:05.744086 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:05.744095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:05.744138 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:50:05.744659 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:05.744668 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:05.744703 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:05.744857 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:05.745495 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:05.745499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:05.745499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:05.745504 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:05.745547 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:05.745560 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:05.745640 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:05.745690 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:50:05.745731 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260810 04:50:05.746670 18941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723: Bootstrap starting.
I20260810 04:50:05.746848 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:05.747311 18941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:05.747632 18941 log.cc:824] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:05.748440 18941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723: No bootstrap required, opened a new log
I20260810 04:50:05.749416 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:05.748338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:57477 (local address 127.0.0.1:45627)
0810 04:50:05.748372 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:05.748374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:05.748492 (+ 118us) server_negotiation.cc:407] Connection header received
0810 04:50:05.748563 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:05.748564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:05.748569 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:05.748611 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:05.749164 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:05.749165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:05.749165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:05.749170 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:05.749181 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:05.749185 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:05.749261 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:05.749342 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 04:50:05.749354 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:50:05.750591 18941 raft_consensus.cc:348] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c472f3814cdd450681e766e6082cf723" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45627 } }
I20260810 04:50:05.750736 18941 raft_consensus.cc:374] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:05.750758 18941 raft_consensus.cc:729] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c472f3814cdd450681e766e6082cf723, State: Initialized, Role: FOLLOWER
I20260810 04:50:05.750839 18941 consensus_queue.cc:260] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c472f3814cdd450681e766e6082cf723" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45627 } }
I20260810 04:50:05.750898 18941 raft_consensus.cc:388] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:05.750919 18941 raft_consensus.cc:482] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:05.750949 18941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:05.751578 18941 raft_consensus.cc:504] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c472f3814cdd450681e766e6082cf723" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45627 } }
I20260810 04:50:05.751674 18941 leader_election.cc:302] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c472f3814cdd450681e766e6082cf723; no voters:
I20260810 04:50:05.751821 18941 leader_election.cc:288] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:05.751900 18953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:05.752046 18953 raft_consensus.cc:686] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [term 1 LEADER]: Becoming Leader. State: Replica: c472f3814cdd450681e766e6082cf723, State: Running, Role: LEADER
I20260810 04:50:05.752072 18941 sys_catalog.cc:564] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:05.752180 18953 consensus_queue.cc:237] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c472f3814cdd450681e766e6082cf723" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45627 } }
I20260810 04:50:05.752576 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c472f3814cdd450681e766e6082cf723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c472f3814cdd450681e766e6082cf723" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45627 } } }
I20260810 04:50:05.752599 18956 sys_catalog.cc:455] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c472f3814cdd450681e766e6082cf723. Latest consensus state: current_term: 1 leader_uuid: "c472f3814cdd450681e766e6082cf723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c472f3814cdd450681e766e6082cf723" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45627 } } }
I20260810 04:50:05.752744 18956 sys_catalog.cc:458] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:05.752944 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:05.753067 18968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:05.753603 18968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:05.755162 18968 catalog_manager.cc:1347] Generated new cluster ID: 246784e47eda41ceb51e3b3de301a650
I20260810 04:50:05.755220 18968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:05.787513 18968 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:05.787765 18968 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:05.797227 18968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c472f3814cdd450681e766e6082cf723: Generated new TSK 0
I20260810 04:50:05.797480 18968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:05.802745 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 18823
2026-08-10T04:50:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1591 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:05 dist-test-slave-8hhm krb5kdc[18987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:05 dist-test-slave-8hhm krb5kdc[18987](info): set up 2 sockets
Aug 10 04:50:05 dist-test-slave-8hhm krb5kdc[18987](info): commencing operation
krb5kdc: starting...
W20260810 04:50:07.089550 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:07 dist-test-slave-8hhm krb5kdc[18987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:50:07.211783 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:07.297190 19010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:07.297482 19010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:07.297521 19010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:07.298655 19010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:07.298731 19010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:07.298753 19010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:07.298774 19010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:07.298803 19010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:07.300267 19010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:07.300544 19010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:07.300771 19010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:07.305787 19023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:07.305809 19021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:07.305809 19019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:07.305956 19010 server_base.cc:1034] running on GCE node
I20260810 04:50:07.306176 19010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:07.306428 19010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:07.307577 19010 hybrid_clock.cc:648] HybridClock initialized: now 1786337407307552 us; error 39 us; skew 500 ppm
Aug 10 04:50:07 dist-test-slave-8hhm krb5kdc[18987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:07.310242 19010 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:50:07.310724 19010 webserver.cc:460] Webserver started at http://127.0.0.1:45467/ using document root <none> and password file <none>
I20260810 04:50:07.310959 19010 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:07.311030 19010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:07.311189 19010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:07.312271 19010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "ebe69e47c95c43968f0c31b7308c60a8"
format_stamp: "Formatted at 2026-08-10 04:50:07 on dist-test-slave-8hhm"
server_key: "63209ed94dfc276a0c528ffeaaeeef29"
server_key_iv: "c4779256b0708d7f4ff5301d6bb13f6b"
server_key_version: "encryptionkey@0"
I20260810 04:50:07.312647 19010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "ebe69e47c95c43968f0c31b7308c60a8"
format_stamp: "Formatted at 2026-08-10 04:50:07 on dist-test-slave-8hhm"
server_key: "63209ed94dfc276a0c528ffeaaeeef29"
server_key_iv: "c4779256b0708d7f4ff5301d6bb13f6b"
server_key_version: "encryptionkey@0"
I20260810 04:50:07.314095 19010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:50:07.314894 19037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:07.315069 19010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:50:07.315205 19010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "ebe69e47c95c43968f0c31b7308c60a8"
format_stamp: "Formatted at 2026-08-10 04:50:07 on dist-test-slave-8hhm"
server_key: "63209ed94dfc276a0c528ffeaaeeef29"
server_key_iv: "c4779256b0708d7f4ff5301d6bb13f6b"
server_key_version: "encryptionkey@0"
I20260810 04:50:07.315286 19010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:07.328637 19010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:07.330461 19010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:07.330629 19010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:07.334834 19010 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35467
I20260810 04:50:07.334865 19131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35467 every 8 connection(s)
I20260810 04:50:07.335222 19010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:07.336061 19132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:07.337299 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19010
I20260810 04:50:07.337422 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:07.337615 15284 external_mini_cluster.cc:1428] Setting key 490ab4f367d60d402678a5d480c4c503
I20260810 04:50:07.338896 19132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8: Bootstrap starting.
I20260810 04:50:07.339687 19132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:07.340021 19132 log.cc:824] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:07.340793 19132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8: No bootstrap required, opened a new log
I20260810 04:50:07.343185 19132 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe69e47c95c43968f0c31b7308c60a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20260810 04:50:07.343374 19132 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:07.343420 19132 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe69e47c95c43968f0c31b7308c60a8, State: Initialized, Role: FOLLOWER
I20260810 04:50:07.343562 19132 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe69e47c95c43968f0c31b7308c60a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20260810 04:50:07.343660 19132 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:07.343709 19132 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:07.343775 19132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:07.344595 19132 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe69e47c95c43968f0c31b7308c60a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20260810 04:50:07.344735 19132 leader_election.cc:302] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe69e47c95c43968f0c31b7308c60a8; no voters:
Aug 10 04:50:07 dist-test-slave-8hhm krb5kdc[18987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:50:07.344944 19132 leader_election.cc:288] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:07.345014 19140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:07.345201 19132 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:07.345342 19140 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [term 1 LEADER]: Becoming Leader. State: Replica: ebe69e47c95c43968f0c31b7308c60a8, State: Running, Role: LEADER
I20260810 04:50:07.345453 19140 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe69e47c95c43968f0c31b7308c60a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20260810 04:50:07.345851 19141 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebe69e47c95c43968f0c31b7308c60a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe69e47c95c43968f0c31b7308c60a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } } }
I20260810 04:50:07.346014 19141 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:07.345841 19140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe69e47c95c43968f0c31b7308c60a8. Latest consensus state: current_term: 1 leader_uuid: "ebe69e47c95c43968f0c31b7308c60a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe69e47c95c43968f0c31b7308c60a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } } }
I20260810 04:50:07.346295 19140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:07.346908 19160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:07.347376 19160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:07.349618 19160 catalog_manager.cc:1347] Generated new cluster ID: d2015231fbbf47d4832a857d4b306c1a
I20260810 04:50:07.349673 19160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:07.349671 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:07.338148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:57619 (local address 127.0.0.1:35467)
0810 04:50:07.338338 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:07.338351 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:07.338376 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:50:07.338794 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:07.338800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:07.338844 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:07.338955 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:07.339829 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:07.341011 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:07.342058 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:07.342445 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:07.342538 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:07.345690 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:07.345695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:07.345695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:07.345706 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:07.347797 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:07.348214 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:07.348224 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:07.348225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:07.348334 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:07.348649 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:07.348650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:07.348650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:07.348948 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:07.349123 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:07.349323 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 04:50:07.349431 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260810 04:50:07.350652 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:07.359813 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:07.352176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:59051 (local address 127.0.0.1:35467)
0810 04:50:07.352210 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:07.352212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:07.352371 (+ 159us) server_negotiation.cc:407] Connection header received
0810 04:50:07.352397 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:07.352398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:07.352409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:07.352477 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:07.353070 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:07.354196 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:07.355197 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:07.355547 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:07.355621 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:07.356915 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:07.356916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:07.356917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:07.356924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:07.358572 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:07.358974 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:07.358975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:07.358975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:07.359035 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:07.359267 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:07.359268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:07.359268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:07.359395 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:07.359499 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:07.359657 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 04:50:07.359722 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:50:07.361608 19160 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:07.362253 19160 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:07.370644 19160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebe69e47c95c43968f0c31b7308c60a8: Generated new TSK 0
I20260810 04:50:07.370827 19160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:07.374526 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19010
W20260810 04:50:07.403551 18982 connection.cc:535] client connection to 127.0.0.1:35467 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35467): Connection reset by peer (error 104)
2026-08-10T04:50:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1586 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:07 dist-test-slave-8hhm krb5kdc[19183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:07 dist-test-slave-8hhm krb5kdc[19183](info): set up 2 sockets
Aug 10 04:50:07 dist-test-slave-8hhm krb5kdc[19183](info): commencing operation
krb5kdc: starting...
W20260810 04:50:08.657094 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:08 dist-test-slave-8hhm krb5kdc[19183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:50:08.777563 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:08.859130 19212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:08.859375 19212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:08.859429 19212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:08.860540 19212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:08.860607 19212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:08.860631 19212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:08.860653 19212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:08.860677 19212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:08.862282 19212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:08.862555 19212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:08.862784 19212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:08.867673 19218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:08.867625 19221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:08.867698 19217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:08.867874 19212 server_base.cc:1034] running on GCE node
I20260810 04:50:08.868099 19212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:08.868330 19212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:08.869463 19212 hybrid_clock.cc:648] HybridClock initialized: now 1786337408869449 us; error 28 us; skew 500 ppm
Aug 10 04:50:08 dist-test-slave-8hhm krb5kdc[19183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:08.871934 19212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:50:08.872319 19212 webserver.cc:460] Webserver started at http://127.0.0.1:36445/ using document root <none> and password file <none>
I20260810 04:50:08.872573 19212 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:08.872619 19212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:08.872710 19212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:08.873785 19212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "2d672e283b3047c88288f3afb33fd657"
format_stamp: "Formatted at 2026-08-10 04:50:08 on dist-test-slave-8hhm"
server_key: "51dc8e35f0c3a04f92be6cf2d42abdde"
server_key_iv: "8f13d06ee7c499530226468d7b715239"
server_key_version: "encryptionkey@0"
I20260810 04:50:08.874081 19212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "2d672e283b3047c88288f3afb33fd657"
format_stamp: "Formatted at 2026-08-10 04:50:08 on dist-test-slave-8hhm"
server_key: "51dc8e35f0c3a04f92be6cf2d42abdde"
server_key_iv: "8f13d06ee7c499530226468d7b715239"
server_key_version: "encryptionkey@0"
I20260810 04:50:08.875344 19212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:50:08.876053 19229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:08.876233 19212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:50:08.876344 19212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "2d672e283b3047c88288f3afb33fd657"
format_stamp: "Formatted at 2026-08-10 04:50:08 on dist-test-slave-8hhm"
server_key: "51dc8e35f0c3a04f92be6cf2d42abdde"
server_key_iv: "8f13d06ee7c499530226468d7b715239"
server_key_version: "encryptionkey@0"
I20260810 04:50:08.876422 19212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:08.895828 19212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:08.897954 19212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:08.898126 19212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:08.902100 19212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37597
I20260810 04:50:08.902166 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37597 every 8 connection(s)
I20260810 04:50:08.902539 19212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:08.903002 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19212
I20260810 04:50:08.903148 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:08.903380 15284 external_mini_cluster.cc:1428] Setting key 7bf6a41fdae98a65b89446d8fe0097f4
I20260810 04:50:08.903669 19319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:08.906953 19319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657: Bootstrap starting.
I20260810 04:50:08.907570 19319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:08.907899 19319 log.cc:824] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:08.908776 19319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657: No bootstrap required, opened a new log
I20260810 04:50:08.909162 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:08.903883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:41541 (local address 127.0.0.1:37597)
0810 04:50:08.904101 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:08.904111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:08.904144 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:50:08.904783 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:08.904790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:08.904841 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:08.904993 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:08.905255 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:08.906161 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:08.907019 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:08.907468 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:08.907614 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:08.908391 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:08.908395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:08.908395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:08.908614 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:08.908673 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:08.908688 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:08.908787 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:08.908914 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 04:50:08.908944 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260810 04:50:08.910172 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:08.911533 19319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d672e283b3047c88288f3afb33fd657" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } }
I20260810 04:50:08.911738 19319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:08.911810 19319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d672e283b3047c88288f3afb33fd657, State: Initialized, Role: FOLLOWER
I20260810 04:50:08.911942 19319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d672e283b3047c88288f3afb33fd657" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } }
I20260810 04:50:08.912040 19319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:08.912081 19319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:08.912149 19319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:08.912811 19319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d672e283b3047c88288f3afb33fd657" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } }
I20260810 04:50:08.912953 19319 leader_election.cc:302] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d672e283b3047c88288f3afb33fd657; no voters:
I20260810 04:50:08.913133 19319 leader_election.cc:288] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:08.913201 19335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:08.913394 19319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:08.913573 19335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [term 1 LEADER]: Becoming Leader. State: Replica: 2d672e283b3047c88288f3afb33fd657, State: Running, Role: LEADER
I20260810 04:50:08.913722 19335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d672e283b3047c88288f3afb33fd657" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } }
I20260810 04:50:08.914125 19337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d672e283b3047c88288f3afb33fd657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d672e283b3047c88288f3afb33fd657" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } } }
I20260810 04:50:08.914215 19337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:08.914389 19338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d672e283b3047c88288f3afb33fd657. Latest consensus state: current_term: 1 leader_uuid: "2d672e283b3047c88288f3afb33fd657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d672e283b3047c88288f3afb33fd657" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37597 } } }
I20260810 04:50:08.914445 19338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:08.914889 19361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:08.915436 19361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:08.915576 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:08.911586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:59941 (local address 127.0.0.1:37597)
0810 04:50:08.911626 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:08.911628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:08.911760 (+ 132us) server_negotiation.cc:407] Connection header received
0810 04:50:08.911842 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:08.911843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:08.911854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:08.911915 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:08.912129 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:08.912811 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:08.913972 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:08.914304 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:08.914355 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:08.915222 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:08.915224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:08.915224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:08.915228 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:08.915243 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:08.915248 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:08.915284 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:08.915424 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 04:50:08.915435 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:50:08.916988 19361 catalog_manager.cc:1347] Generated new cluster ID: 712bdd1a55524706b52797b231a9ed79
I20260810 04:50:08.917039 19361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:08.924235 19361 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:08.924752 19361 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:08.927662 19361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d672e283b3047c88288f3afb33fd657: Generated new TSK 0
I20260810 04:50:08.927832 19361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:08.930258 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19212
W20260810 04:50:08.948786 19181 connection.cc:535] client connection to 127.0.0.1:37597 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37597): Connection reset by peer (error 104)
2026-08-10T04:50:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1545 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:09 dist-test-slave-8hhm krb5kdc[19385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:09 dist-test-slave-8hhm krb5kdc[19385](info): set up 2 sockets
Aug 10 04:50:09 dist-test-slave-8hhm krb5kdc[19385](info): commencing operation
krb5kdc: starting...
W20260810 04:50:10.236579 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:10 dist-test-slave-8hhm krb5kdc[19385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:50:10.356588 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:10.436241 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:10.436439 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:10.436479 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:10.437593 19406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:10.437630 19406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:10.437644 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:10.437656 19406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:10.437668 19406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:10.439051 19406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:10.439240 19406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:10.439399 19406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:10.444252 19420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:10.444259 19416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:10.444576 19406 server_base.cc:1034] running on GCE node
W20260810 04:50:10.444299 19415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:10.444873 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:10.445166 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:10.446398 19406 hybrid_clock.cc:648] HybridClock initialized: now 1786337410446372 us; error 39 us; skew 500 ppm
Aug 10 04:50:10 dist-test-slave-8hhm krb5kdc[19385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:10.448994 19406 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:50:10.449479 19406 webserver.cc:460] Webserver started at http://127.0.0.1:42475/ using document root <none> and password file <none>
I20260810 04:50:10.449714 19406 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:10.449765 19406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:10.449930 19406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:10.450989 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "834211dff9cd47ff89607586a7928c35"
format_stamp: "Formatted at 2026-08-10 04:50:10 on dist-test-slave-8hhm"
server_key: "da3f6cc4fe2049c211361b8655b058a8"
server_key_iv: "c0d02b4a0e10b2411a6e4e54b38495a5"
server_key_version: "encryptionkey@0"
I20260810 04:50:10.451411 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "834211dff9cd47ff89607586a7928c35"
format_stamp: "Formatted at 2026-08-10 04:50:10 on dist-test-slave-8hhm"
server_key: "da3f6cc4fe2049c211361b8655b058a8"
server_key_iv: "c0d02b4a0e10b2411a6e4e54b38495a5"
server_key_version: "encryptionkey@0"
I20260810 04:50:10.452762 19406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:50:10.453615 19433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:10.453866 19406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:50:10.453938 19406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "834211dff9cd47ff89607586a7928c35"
format_stamp: "Formatted at 2026-08-10 04:50:10 on dist-test-slave-8hhm"
server_key: "da3f6cc4fe2049c211361b8655b058a8"
server_key_iv: "c0d02b4a0e10b2411a6e4e54b38495a5"
server_key_version: "encryptionkey@0"
I20260810 04:50:10.454023 19406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:10.457265 19406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:10.459666 19406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:10.459817 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:10.463871 19406 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46557
I20260810 04:50:10.463903 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46557 every 8 connection(s)
I20260810 04:50:10.464205 19406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:10.465276 19507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:10.467474 19507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35: Bootstrap starting.
I20260810 04:50:10.468029 19507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:10.468321 19507 log.cc:824] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:10.468969 19507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35: No bootstrap required, opened a new log
I20260810 04:50:10.470580 19507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834211dff9cd47ff89607586a7928c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46557 } }
I20260810 04:50:10.470706 19507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:10.470727 19507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834211dff9cd47ff89607586a7928c35, State: Initialized, Role: FOLLOWER
I20260810 04:50:10.470863 19507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834211dff9cd47ff89607586a7928c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46557 } }
I20260810 04:50:10.470942 19507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:10.470963 19507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:10.470990 19507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:10.471457 19507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834211dff9cd47ff89607586a7928c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46557 } }
I20260810 04:50:10.471530 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19406
I20260810 04:50:10.471573 19507 leader_election.cc:302] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 834211dff9cd47ff89607586a7928c35; no voters:
I20260810 04:50:10.471684 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:10.471786 19507 leader_election.cc:288] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:10.471818 19513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:10.471933 15284 external_mini_cluster.cc:1428] Setting key f01546eed40a63e83b1c31ac7f9a7282
I20260810 04:50:10.472018 19513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [term 1 LEADER]: Becoming Leader. State: Replica: 834211dff9cd47ff89607586a7928c35, State: Running, Role: LEADER
I20260810 04:50:10.472126 19507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:10.472121 19513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834211dff9cd47ff89607586a7928c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46557 } }
I20260810 04:50:10.472432 19513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 834211dff9cd47ff89607586a7928c35. Latest consensus state: current_term: 1 leader_uuid: "834211dff9cd47ff89607586a7928c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834211dff9cd47ff89607586a7928c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46557 } } }
I20260810 04:50:10.472494 19513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:10.472582 19515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "834211dff9cd47ff89607586a7928c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834211dff9cd47ff89607586a7928c35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46557 } } }
I20260810 04:50:10.472674 19515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:10.473629 19523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:10.474320 19523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:10.474767 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:10.472576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:47811 (local address 127.0.0.1:46557)
0810 04:50:10.472732 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:10.472742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:10.473021 (+ 279us) server_negotiation.cc:407] Connection header received
0810 04:50:10.473332 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:10.473340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:10.473374 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:10.473762 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:10.474186 (+ 424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:10.474189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:10.474190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:10.474194 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:10.474244 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:10.474262 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:10.474333 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:10.474432 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 04:50:10.474596 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260810 04:50:10.475723 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:10.475988 19523 catalog_manager.cc:1347] Generated new cluster ID: 0583b3cec5064a10b9429523faccc3c4
I20260810 04:50:10.476032 19523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:10.482326 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:10.477185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.202:46685 (local address 127.0.0.1:46557)
0810 04:50:10.477265 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:10.477267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:10.481413 (+ 4146us) server_negotiation.cc:407] Connection header received
0810 04:50:10.481474 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:10.481475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:10.481481 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:10.481541 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:10.482105 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:10.482106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:10.482106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:10.482110 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:10.482123 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:10.482127 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:10.482178 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:10.482247 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 04:50:10.482258 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:50:10.485852 19523 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:10.485991 19523 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:10.490267 19523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 834211dff9cd47ff89607586a7928c35: Generated new TSK 0
I20260810 04:50:10.490455 19523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:10.492892 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19406
2026-08-10T04:50:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1563 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:10 dist-test-slave-8hhm krb5kdc[19558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:10 dist-test-slave-8hhm krb5kdc[19558](info): set up 2 sockets
Aug 10 04:50:10 dist-test-slave-8hhm krb5kdc[19558](info): commencing operation
krb5kdc: starting...
W20260810 04:50:11.760133 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:11 dist-test-slave-8hhm krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:50:11.879276 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37847
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:36927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:37847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:11.959715 19581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:11.959942 19581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:11.959978 19581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:11.961069 19581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:11.961141 19581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:11.961164 19581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:11.961185 19581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:11.961237 19581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:11.962673 19581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:36927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:37847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:37847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:11.962953 19581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:11.963172 19581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:11.968063 19591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:11.968070 19596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:11.968070 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:11.968288 19581 server_base.cc:1034] running on GCE node
I20260810 04:50:11.968537 19581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:11.968834 19581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:11.970010 19581 hybrid_clock.cc:648] HybridClock initialized: now 1786337411969991 us; error 45 us; skew 500 ppm
Aug 10 04:50:11 dist-test-slave-8hhm krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337411, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:11.975867 19581 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:50:11.976389 19581 webserver.cc:460] Webserver started at http://127.14.237.62:44659/ using document root <none> and password file <none>
I20260810 04:50:11.976615 19581 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:11.976680 19581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:11.976830 19581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:11.978004 19581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "b04ac8770e2140deb880bbe45d2f1ae9"
format_stamp: "Formatted at 2026-08-10 04:50:11 on dist-test-slave-8hhm"
server_key: "ed8b9ff36d1c7155f8820ecce0a008fa"
server_key_iv: "9c79dd7bcc5dcd319d99b0b516bff4f6"
server_key_version: "encryptionkey@0"
I20260810 04:50:11.978466 19581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "b04ac8770e2140deb880bbe45d2f1ae9"
format_stamp: "Formatted at 2026-08-10 04:50:11 on dist-test-slave-8hhm"
server_key: "ed8b9ff36d1c7155f8820ecce0a008fa"
server_key_iv: "9c79dd7bcc5dcd319d99b0b516bff4f6"
server_key_version: "encryptionkey@0"
I20260810 04:50:11.979753 19581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:50:11.980508 19607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:11.980702 19581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:50:11.980814 19581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "b04ac8770e2140deb880bbe45d2f1ae9"
format_stamp: "Formatted at 2026-08-10 04:50:11 on dist-test-slave-8hhm"
server_key: "ed8b9ff36d1c7155f8820ecce0a008fa"
server_key_iv: "9c79dd7bcc5dcd319d99b0b516bff4f6"
server_key_version: "encryptionkey@0"
I20260810 04:50:11.980890 19581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:11.990146 19581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:11.992375 19581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:11.992530 19581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:11.996726 19581 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:37847
I20260810 04:50:11.996763 19688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:37847 every 8 connection(s)
I20260810 04:50:11.997089 19581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:11.997982 19690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:12.000097 19690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9: Bootstrap starting.
I20260810 04:50:12.000641 19690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:12.000891 19690 log.cc:824] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:12.001593 19690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9: No bootstrap required, opened a new log
I20260810 04:50:12.003053 19690 raft_consensus.cc:348] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ac8770e2140deb880bbe45d2f1ae9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37847 } }
I20260810 04:50:12.003203 19690 raft_consensus.cc:374] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:12.003225 19690 raft_consensus.cc:729] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04ac8770e2140deb880bbe45d2f1ae9, State: Initialized, Role: FOLLOWER
I20260810 04:50:12.003324 19690 consensus_queue.cc:260] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ac8770e2140deb880bbe45d2f1ae9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37847 } }
I20260810 04:50:12.003412 19690 raft_consensus.cc:388] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:12.003451 19690 raft_consensus.cc:482] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:12.003501 19690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:12.003952 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19581
I20260810 04:50:12.004101 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:12.004344 15284 external_mini_cluster.cc:1428] Setting key c7a1b5d947365b7fd2a824e6ca8a22d0
I20260810 04:50:12.004643 19690 raft_consensus.cc:504] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ac8770e2140deb880bbe45d2f1ae9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37847 } }
I20260810 04:50:12.004801 19690 leader_election.cc:302] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b04ac8770e2140deb880bbe45d2f1ae9; no voters:
I20260810 04:50:12.004978 19690 leader_election.cc:288] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:12.005029 19694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:12.005199 19694 raft_consensus.cc:686] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [term 1 LEADER]: Becoming Leader. State: Replica: b04ac8770e2140deb880bbe45d2f1ae9, State: Running, Role: LEADER
I20260810 04:50:12.005267 19690 sys_catalog.cc:564] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:12.005344 19694 consensus_queue.cc:237] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ac8770e2140deb880bbe45d2f1ae9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37847 } }
I20260810 04:50:12.005689 19697 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b04ac8770e2140deb880bbe45d2f1ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ac8770e2140deb880bbe45d2f1ae9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37847 } } }
I20260810 04:50:12.005678 19694 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04ac8770e2140deb880bbe45d2f1ae9. Latest consensus state: current_term: 1 leader_uuid: "b04ac8770e2140deb880bbe45d2f1ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04ac8770e2140deb880bbe45d2f1ae9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 37847 } } }
I20260810 04:50:12.005837 19694 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:12.006114 19697 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:12.006492 19704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:12.006971 19704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:12.008641 19704 catalog_manager.cc:1347] Generated new cluster ID: 64d69a31e23f48f99e75245b2d0a150e
I20260810 04:50:12.008698 19704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:50:12 dist-test-slave-8hhm krb5kdc[19558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:50:12.015507 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:12.004861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42035 (local address 127.14.237.62:37847)
0810 04:50:12.005142 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:12.005151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:12.005180 (+ 29us) server_negotiation.cc:407] Connection header received
0810 04:50:12.005325 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:12.005333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:12.005390 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:12.006415 (+ 1025us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:12.006544 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:12.007983 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:12.008951 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:12.009430 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:12.009540 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:12.011943 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:12.011946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:12.011947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:12.011958 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:12.013724 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:12.014225 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:12.014228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:12.014228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:12.014367 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:12.014620 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:12.014621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:12.014621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:12.014882 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:12.015074 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:12.015198 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 04:50:12.015306 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
I20260810 04:50:12.016511 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:12.022207 19704 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:12.022866 19704 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:12.027526 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:12.021147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57787 (local address 127.14.237.62:37847)
0810 04:50:12.021185 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:12.021186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:12.021271 (+ 85us) server_negotiation.cc:407] Connection header received
0810 04:50:12.021340 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:12.021341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:12.021361 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:12.021402 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:12.021965 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:12.022588 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:12.023359 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:12.023586 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:12.023662 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:12.024815 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:12.024816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:12.024817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:12.024824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:12.026272 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:12.026638 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:12.026639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:12.026639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:12.026698 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:12.026981 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:12.026982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:12.026982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:12.027110 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:12.027235 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:12.027358 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 04:50:12.027422 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:50:12.029727 19704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b04ac8770e2140deb880bbe45d2f1ae9: Generated new TSK 0
I20260810 04:50:12.029908 19704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:12.032946 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19581
W20260810 04:50:12.048032 19553 connection.cc:535] client connection to 127.14.237.62:37847 recv error: Network error: failed to read from TLS socket (remote: 127.14.237.62:37847): Connection reset by peer (error 104)
2026-08-10T04:50:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1535 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:12 dist-test-slave-8hhm krb5kdc[19735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:12 dist-test-slave-8hhm krb5kdc[19735](info): set up 2 sockets
Aug 10 04:50:12 dist-test-slave-8hhm krb5kdc[19735](info): commencing operation
krb5kdc: starting...
W20260810 04:50:13.276561 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:13 dist-test-slave-8hhm krb5kdc[19735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:50:13.394212 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:43369
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:43369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:13.472885 19760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:13.473083 19760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:13.473104 19760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:13.474167 19760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:13.474228 19760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:13.474242 19760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:13.474254 19760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:13.474306 19760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:13.475786 19760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:43369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:13.475979 19760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:13.476207 19760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:13.481205 19772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:13.481261 19776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:13.481439 19760 server_base.cc:1034] running on GCE node
W20260810 04:50:13.481287 19771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:13.481760 19760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:13.482044 19760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:13.483220 19760 hybrid_clock.cc:648] HybridClock initialized: now 1786337413483202 us; error 31 us; skew 500 ppm
Aug 10 04:50:13 dist-test-slave-8hhm krb5kdc[19735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337413, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:13.485699 19760 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:50:13.486111 19760 webserver.cc:460] Webserver started at http://127.14.237.62:44429/ using document root <none> and password file <none>
I20260810 04:50:13.486352 19760 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:13.486423 19760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:13.486577 19760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:13.487653 19760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8"
format_stamp: "Formatted at 2026-08-10 04:50:13 on dist-test-slave-8hhm"
server_key: "d26313ab7dba16bdab23abaf078ca885"
server_key_iv: "7ca4c25155768c7033e1fbf3d8a3928b"
server_key_version: "encryptionkey@0"
I20260810 04:50:13.488008 19760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8"
format_stamp: "Formatted at 2026-08-10 04:50:13 on dist-test-slave-8hhm"
server_key: "d26313ab7dba16bdab23abaf078ca885"
server_key_iv: "7ca4c25155768c7033e1fbf3d8a3928b"
server_key_version: "encryptionkey@0"
I20260810 04:50:13.489497 19760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:50:13.490182 19786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:13.490351 19760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:13.490455 19760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8"
format_stamp: "Formatted at 2026-08-10 04:50:13 on dist-test-slave-8hhm"
server_key: "d26313ab7dba16bdab23abaf078ca885"
server_key_iv: "7ca4c25155768c7033e1fbf3d8a3928b"
server_key_version: "encryptionkey@0"
I20260810 04:50:13.490528 19760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:13.500312 19760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:13.502250 19760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:13.502404 19760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:13.506445 19760 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:43369
I20260810 04:50:13.506486 19870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:43369 every 8 connection(s)
I20260810 04:50:13.506825 19760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:13.507706 19872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:13.508898 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19760
I20260810 04:50:13.509017 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:13.509198 15284 external_mini_cluster.cc:1428] Setting key f849398157903c97810981852da682af
I20260810 04:50:13.510238 19872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8: Bootstrap starting.
I20260810 04:50:13.510941 19872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:13.511229 19872 log.cc:824] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:13.511983 19872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8: No bootstrap required, opened a new log
I20260810 04:50:13.514689 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:13.509739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59617 (local address 127.14.237.62:43369)
0810 04:50:13.509917 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:13.509924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:13.509947 (+ 23us) server_negotiation.cc:407] Connection header received
0810 04:50:13.510323 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:13.510349 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:13.510420 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:13.510524 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:13.510753 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:13.511662 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:13.512748 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:13.513146 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:13.513282 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:13.514018 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:13.514022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:13.514022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:13.514173 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:13.514201 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:13.514212 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:13.514287 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:13.514442 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 04:50:13.514493 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260810 04:50:13.514946 19872 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43369 } }
I20260810 04:50:13.515173 19872 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:13.515219 19872 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e24e7c19b04fd4b5acc3287a7b9df8, State: Initialized, Role: FOLLOWER
I20260810 04:50:13.515403 19872 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43369 } }
I20260810 04:50:13.515501 19872 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:13.515547 19872 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:13.515596 19872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:13.515763 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:13.516217 19872 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43369 } }
I20260810 04:50:13.516335 19872 leader_election.cc:302] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5e24e7c19b04fd4b5acc3287a7b9df8; no voters:
I20260810 04:50:13.516489 19872 leader_election.cc:288] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:13.516515 19885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:13.516641 19885 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [term 1 LEADER]: Becoming Leader. State: Replica: f5e24e7c19b04fd4b5acc3287a7b9df8, State: Running, Role: LEADER
I20260810 04:50:13.516746 19872 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:13.516778 19885 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43369 } }
I20260810 04:50:13.517112 19885 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e24e7c19b04fd4b5acc3287a7b9df8. Latest consensus state: current_term: 1 leader_uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43369 } } }
I20260810 04:50:13.517246 19885 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:13.517102 19887 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e24e7c19b04fd4b5acc3287a7b9df8" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 43369 } } }
I20260810 04:50:13.517484 19887 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:13.517843 19901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:13.518452 19901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:13.519969 19901 catalog_manager.cc:1347] Generated new cluster ID: 3984b08918794933bf24ae94a4cc82fd
I20260810 04:50:13.520018 19901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:13.521068 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:13.517205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57871 (local address 127.14.237.62:43369)
0810 04:50:13.517283 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:13.517284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:13.517470 (+ 186us) server_negotiation.cc:407] Connection header received
0810 04:50:13.517678 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:13.517679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:13.517690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:13.517773 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:13.517939 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:13.518643 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:13.519660 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:13.519931 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:13.519993 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:13.520714 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:13.520716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:13.520716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:13.520720 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:13.520733 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:13.520739 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:13.520789 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:13.520997 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 04:50:13.521006 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 04:50:13.540390 19901 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:13.540947 19901 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:13.545246 19901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5e24e7c19b04fd4b5acc3287a7b9df8: Generated new TSK 0
I20260810 04:50:13.545558 19901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:13.548238 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19760
W20260810 04:50:13.563624 19733 connection.cc:535] client connection to 127.14.237.62:43369 recv error: Network error: failed to read from TLS socket (remote: 127.14.237.62:43369): Connection reset by peer (error 104)
2026-08-10T04:50:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1515 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:13 dist-test-slave-8hhm krb5kdc[19925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:13 dist-test-slave-8hhm krb5kdc[19925](info): set up 2 sockets
Aug 10 04:50:13 dist-test-slave-8hhm krb5kdc[19925](info): commencing operation
krb5kdc: starting...
W20260810 04:50:14.806488 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:14 dist-test-slave-8hhm krb5kdc[19925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:50:14.928926 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:44273
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:38941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:44273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:15.012270 19944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:15.012475 19944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:15.012497 19944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:15.013705 19944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:15.013746 19944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:15.013759 19944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:15.013772 19944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:15.013783 19944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:15.015175 19944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:38941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:44273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:44273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:15.015367 19944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:15.015528 19944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:15.020454 19955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:15.020483 19951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:15.020470 19952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:15.020784 19944 server_base.cc:1034] running on GCE node
I20260810 04:50:15.021030 19944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:15.021379 19944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:15.022533 19944 hybrid_clock.cc:648] HybridClock initialized: now 1786337415022515 us; error 36 us; skew 500 ppm
Aug 10 04:50:15 dist-test-slave-8hhm krb5kdc[19925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337415, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:15.025085 19944 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:50:15.025523 19944 webserver.cc:460] Webserver started at http://127.14.237.62:40671/ using document root <none> and password file <none>
I20260810 04:50:15.025745 19944 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:15.025816 19944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:15.025969 19944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:15.027084 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "6313be28d5bb404eb606f6c4dba0699a"
format_stamp: "Formatted at 2026-08-10 04:50:15 on dist-test-slave-8hhm"
server_key: "a97ddb368a15bb8a67a6245f0bea7ced"
server_key_iv: "d50dc3bc37848c76109f39360b57bf61"
server_key_version: "encryptionkey@0"
I20260810 04:50:15.027451 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "6313be28d5bb404eb606f6c4dba0699a"
format_stamp: "Formatted at 2026-08-10 04:50:15 on dist-test-slave-8hhm"
server_key: "a97ddb368a15bb8a67a6245f0bea7ced"
server_key_iv: "d50dc3bc37848c76109f39360b57bf61"
server_key_version: "encryptionkey@0"
I20260810 04:50:15.028941 19944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:50:15.029774 19971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:15.029970 19944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:50:15.030071 19944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "6313be28d5bb404eb606f6c4dba0699a"
format_stamp: "Formatted at 2026-08-10 04:50:15 on dist-test-slave-8hhm"
server_key: "a97ddb368a15bb8a67a6245f0bea7ced"
server_key_iv: "d50dc3bc37848c76109f39360b57bf61"
server_key_version: "encryptionkey@0"
I20260810 04:50:15.030184 19944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:15.033496 19944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:15.035311 19944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:15.035462 19944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:15.039292 19944 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:44273
I20260810 04:50:15.039402 20059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:44273 every 8 connection(s)
I20260810 04:50:15.039675 19944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:15.040705 20060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:15.042922 20060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a: Bootstrap starting.
I20260810 04:50:15.043431 20060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:15.043663 20060 log.cc:824] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:15.043955 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 19944
I20260810 04:50:15.044085 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:15.044296 15284 external_mini_cluster.cc:1428] Setting key 8357f11ca03f91a04d8c0e7521c056c7
I20260810 04:50:15.044432 20060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a: No bootstrap required, opened a new log
I20260810 04:50:15.046972 20060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6313be28d5bb404eb606f6c4dba0699a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44273 } }
I20260810 04:50:15.047122 20060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:15.047147 20060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6313be28d5bb404eb606f6c4dba0699a, State: Initialized, Role: FOLLOWER
I20260810 04:50:15.047283 20060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6313be28d5bb404eb606f6c4dba0699a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44273 } }
I20260810 04:50:15.047322 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:15.044867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55633 (local address 127.14.237.62:44273)
0810 04:50:15.045035 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:15.045043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:15.045099 (+ 56us) server_negotiation.cc:407] Connection header received
0810 04:50:15.045855 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:15.045862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:15.045895 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:15.046005 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:15.046649 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:15.046652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:15.046652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:15.046656 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:15.046690 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:15.046700 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:15.046754 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:15.046845 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 04:50:15.046899 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260810 04:50:15.047384 20060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:15.047435 20060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:15.047484 20060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:15.048116 20060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6313be28d5bb404eb606f6c4dba0699a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44273 } }
I20260810 04:50:15.048208 20060 leader_election.cc:302] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6313be28d5bb404eb606f6c4dba0699a; no voters:
I20260810 04:50:15.048331 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:15.048374 20060 leader_election.cc:288] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:15.048403 20070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:15.048559 20070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [term 1 LEADER]: Becoming Leader. State: Replica: 6313be28d5bb404eb606f6c4dba0699a, State: Running, Role: LEADER
I20260810 04:50:15.048686 20060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:15.048702 20070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6313be28d5bb404eb606f6c4dba0699a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44273 } }
I20260810 04:50:15.049149 20071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6313be28d5bb404eb606f6c4dba0699a. Latest consensus state: current_term: 1 leader_uuid: "6313be28d5bb404eb606f6c4dba0699a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6313be28d5bb404eb606f6c4dba0699a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44273 } } }
I20260810 04:50:15.049278 20071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:15.049443 20072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6313be28d5bb404eb606f6c4dba0699a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6313be28d5bb404eb606f6c4dba0699a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 44273 } } }
I20260810 04:50:15.049574 20072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a [sys.catalog]: This master's current role is: LEADER
W20260810 04:50:15.050877 20115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 04:50:15.051028 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:15.049895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48373 (local address 127.14.237.62:44273)
0810 04:50:15.049930 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:15.049931 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:15.050070 (+ 139us) server_negotiation.cc:407] Connection header received
0810 04:50:15.050149 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:15.050150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:15.050155 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:15.050198 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:15.050799 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:15.050800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:15.050801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:15.050805 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:15.050816 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:15.050821 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:15.050869 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:15.050929 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 04:50:15.050943 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 04:50:15.051066 20115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:50:15.051151 20080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:15.051309 20080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:15.052984 20080 catalog_manager.cc:1347] Generated new cluster ID: 03f34e433afa4371b8326f473b39f2d6
I20260810 04:50:15.053040 20080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:15.058176 20080 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:15.058354 20080 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:15.062821 20080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6313be28d5bb404eb606f6c4dba0699a: Generated new TSK 0
I20260810 04:50:15.062997 20080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:15.069764 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 19944
2026-08-10T04:50:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:15 dist-test-slave-8hhm krb5kdc[20125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:15 dist-test-slave-8hhm krb5kdc[20125](info): set up 2 sockets
Aug 10 04:50:15 dist-test-slave-8hhm krb5kdc[20125](info): commencing operation
krb5kdc: starting...
W20260810 04:50:16.347616 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:16 dist-test-slave-8hhm krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:50:16.468003 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:45273
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:16.551646 20147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:16.551878 20147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:16.551901 20147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:16.553117 20147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:16.553157 20147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:16.553172 20147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:16.553185 20147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:16.553198 20147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:16.554693 20147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:16.554885 20147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:16.555060 20147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:16.560189 20158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:16.560218 20159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:16.560213 20166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:16.560281 20147 server_base.cc:1034] running on GCE node
I20260810 04:50:16.560669 20147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:16.560998 20147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:16.562182 20147 hybrid_clock.cc:648] HybridClock initialized: now 1786337416562159 us; error 34 us; skew 500 ppm
Aug 10 04:50:16 dist-test-slave-8hhm krb5kdc[20125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337416, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:16.564687 20147 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:50:16.565136 20147 webserver.cc:460] Webserver started at http://127.14.237.62:36279/ using document root <none> and password file <none>
I20260810 04:50:16.565402 20147 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:16.565474 20147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:16.565627 20147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:16.566655 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "45ff957133474b4f8f78241d24f81d3b"
format_stamp: "Formatted at 2026-08-10 04:50:16 on dist-test-slave-8hhm"
server_key: "9e6c634b7efdc0d2688e5ec719ffeca9"
server_key_iv: "4aadab370b308a4a39164dc19983f9b7"
server_key_version: "encryptionkey@0"
I20260810 04:50:16.567025 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "45ff957133474b4f8f78241d24f81d3b"
format_stamp: "Formatted at 2026-08-10 04:50:16 on dist-test-slave-8hhm"
server_key: "9e6c634b7efdc0d2688e5ec719ffeca9"
server_key_iv: "4aadab370b308a4a39164dc19983f9b7"
server_key_version: "encryptionkey@0"
I20260810 04:50:16.568329 20147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:50:16.569012 20176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:16.569181 20147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:50:16.569312 20147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "45ff957133474b4f8f78241d24f81d3b"
format_stamp: "Formatted at 2026-08-10 04:50:16 on dist-test-slave-8hhm"
server_key: "9e6c634b7efdc0d2688e5ec719ffeca9"
server_key_iv: "4aadab370b308a4a39164dc19983f9b7"
server_key_version: "encryptionkey@0"
I20260810 04:50:16.569401 20147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:16.590517 20147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:16.592546 20147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:16.592744 20147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:16.596923 20147 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:45273
I20260810 04:50:16.596961 20264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:45273 every 8 connection(s)
I20260810 04:50:16.597363 20147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:16.598292 20265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:16.600523 20265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b: Bootstrap starting.
I20260810 04:50:16.601048 20265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:16.601395 20265 log.cc:824] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:16.602069 20265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b: No bootstrap required, opened a new log
I20260810 04:50:16.603025 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20147
I20260810 04:50:16.603148 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:16.603358 15284 external_mini_cluster.cc:1428] Setting key b446496154d7eaf842a474ed33d5c683
I20260810 04:50:16.603828 20265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ff957133474b4f8f78241d24f81d3b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45273 } }
I20260810 04:50:16.604053 20265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:16.604110 20265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ff957133474b4f8f78241d24f81d3b, State: Initialized, Role: FOLLOWER
I20260810 04:50:16.604270 20265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ff957133474b4f8f78241d24f81d3b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45273 } }
I20260810 04:50:16.604349 20265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:16.604372 20265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:16.604441 20265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:16.604969 20265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ff957133474b4f8f78241d24f81d3b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45273 } }
I20260810 04:50:16.605058 20265 leader_election.cc:302] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45ff957133474b4f8f78241d24f81d3b; no voters:
I20260810 04:50:16.605257 20265 leader_election.cc:288] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:16.605283 20277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:16.605394 20277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [term 1 LEADER]: Becoming Leader. State: Replica: 45ff957133474b4f8f78241d24f81d3b, State: Running, Role: LEADER
I20260810 04:50:16.605532 20277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ff957133474b4f8f78241d24f81d3b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45273 } }
I20260810 04:50:16.605533 20265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:16.605962 20278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45ff957133474b4f8f78241d24f81d3b. Latest consensus state: current_term: 1 leader_uuid: "45ff957133474b4f8f78241d24f81d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ff957133474b4f8f78241d24f81d3b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45273 } } }
I20260810 04:50:16.606041 20278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:16.606295 20282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45ff957133474b4f8f78241d24f81d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ff957133474b4f8f78241d24f81d3b" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 45273 } } }
I20260810 04:50:16.606572 20282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:16.606714 20285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:16.607254 20285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:16.608966 20285 catalog_manager.cc:1347] Generated new cluster ID: 7b459c4179da4b4f81382ba5bc2684ee
I20260810 04:50:16.609030 20285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:50:16 dist-test-slave-8hhm krb5kdc[20125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:50:16.615989 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:16.603873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53199 (local address 127.14.237.62:45273)
0810 04:50:16.604059 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:16.604066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:16.604091 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:50:16.604274 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:16.604281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:16.604336 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:16.604487 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:16.605682 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:16.607136 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:16.608099 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:16.608643 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:16.608771 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:16.611576 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:16.611580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:16.611581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:16.611593 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:16.613672 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:16.614060 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:16.614063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:16.614063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:16.614817 (+ 754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:16.615032 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:16.615062 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:16.615063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:16.615388 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:16.615575 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:16.615646 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:50:16.615788 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260810 04:50:16.617170 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:16.625478 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:16.618630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39571 (local address 127.14.237.62:45273)
0810 04:50:16.618664 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:16.618666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:16.618760 (+ 94us) server_negotiation.cc:407] Connection header received
0810 04:50:16.618803 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:16.618804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:16.618814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:16.618871 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:16.619427 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:16.620193 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:16.621129 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:16.621467 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:16.621535 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:16.622774 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:16.622775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:16.622776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:16.622783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:16.624327 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:16.624689 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:16.624690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:16.624690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:16.624749 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:16.624940 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:16.624940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:16.624941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:16.625082 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:16.625189 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:16.625312 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 04:50:16.625388 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:50:16.627439 20285 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:16.628091 20285 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:16.632823 20285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45ff957133474b4f8f78241d24f81d3b: Generated new TSK 0
I20260810 04:50:16.633004 20285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:16.640013 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20147
2026-08-10T04:50:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:16 dist-test-slave-8hhm krb5kdc[20321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:16 dist-test-slave-8hhm krb5kdc[20321](info): set up 2 sockets
Aug 10 04:50:16 dist-test-slave-8hhm krb5kdc[20321](info): commencing operation
krb5kdc: starting...
W20260810 04:50:17.899636 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:17 dist-test-slave-8hhm krb5kdc[20321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:50:18.019321 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:33803
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:18.099347 20343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:18.099534 20343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:18.099553 20343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:18.100647 20343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:18.100689 20343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:18.100703 20343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:18.100715 20343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:18.100728 20343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:18.102169 20343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:18.102334 20343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:18.102571 20343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:18.107352 20354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:18.107344 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:18.107522 20343 server_base.cc:1034] running on GCE node
W20260810 04:50:18.107347 20357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:18.107836 20343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:18.108084 20343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:18.109262 20343 hybrid_clock.cc:648] HybridClock initialized: now 1786337418109246 us; error 29 us; skew 500 ppm
Aug 10 04:50:18 dist-test-slave-8hhm krb5kdc[20321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337418, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:18.116665 20343 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:50:18.117097 20343 webserver.cc:460] Webserver started at http://127.14.237.62:39173/ using document root <none> and password file <none>
I20260810 04:50:18.117358 20343 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:18.117404 20343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:18.117564 20343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:18.118551 20343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "8cb40312b04246f185c5527d72bce45d"
format_stamp: "Formatted at 2026-08-10 04:50:18 on dist-test-slave-8hhm"
server_key: "58e4216d9b6267906e0ea968bc458496"
server_key_iv: "a9ecb05b156cd9757ff936eb36f65b4d"
server_key_version: "encryptionkey@0"
I20260810 04:50:18.118948 20343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "8cb40312b04246f185c5527d72bce45d"
format_stamp: "Formatted at 2026-08-10 04:50:18 on dist-test-slave-8hhm"
server_key: "58e4216d9b6267906e0ea968bc458496"
server_key_iv: "a9ecb05b156cd9757ff936eb36f65b4d"
server_key_version: "encryptionkey@0"
I20260810 04:50:18.120343 20343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:50:18.121142 20366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:18.121371 20343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:50:18.121471 20343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "8cb40312b04246f185c5527d72bce45d"
format_stamp: "Formatted at 2026-08-10 04:50:18 on dist-test-slave-8hhm"
server_key: "58e4216d9b6267906e0ea968bc458496"
server_key_iv: "a9ecb05b156cd9757ff936eb36f65b4d"
server_key_version: "encryptionkey@0"
I20260810 04:50:18.121551 20343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:18.136475 20343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:18.138290 20343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:18.138473 20343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:18.142390 20343 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:33803
I20260810 04:50:18.142422 20443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:33803 every 8 connection(s)
I20260810 04:50:18.142751 20343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:18.143599 20444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:18.144006 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20343
I20260810 04:50:18.144134 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:18.144321 15284 external_mini_cluster.cc:1428] Setting key 72ce0b47b1484dba44248342966faebc
I20260810 04:50:18.146731 20444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d: Bootstrap starting.
I20260810 04:50:18.147395 20444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:18.147675 20444 log.cc:824] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:18.148360 20444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d: No bootstrap required, opened a new log
I20260810 04:50:18.150154 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:18.144857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58145 (local address 127.14.237.62:33803)
0810 04:50:18.145059 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:18.145069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:18.145106 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:50:18.145664 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:18.145673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:18.145733 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:18.145885 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:18.146034 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:18.147359 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:18.148145 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:18.148713 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:18.148823 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:18.149459 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:18.149464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:18.149464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:18.149692 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:18.149728 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:18.149745 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:18.149810 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:18.149875 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 04:50:18.149938 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260810 04:50:18.150931 20444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb40312b04246f185c5527d72bce45d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33803 } }
I20260810 04:50:18.151114 20444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:18.151151 20444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb40312b04246f185c5527d72bce45d, State: Initialized, Role: FOLLOWER
I20260810 04:50:18.151273 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:18.151288 20444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb40312b04246f185c5527d72bce45d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33803 } }
I20260810 04:50:18.151392 20444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:18.151432 20444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:18.151482 20444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:18.152004 20444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb40312b04246f185c5527d72bce45d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33803 } }
I20260810 04:50:18.152109 20444 leader_election.cc:302] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cb40312b04246f185c5527d72bce45d; no voters:
I20260810 04:50:18.152314 20444 leader_election.cc:288] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:18.152365 20458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:18.152524 20458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [term 1 LEADER]: Becoming Leader. State: Replica: 8cb40312b04246f185c5527d72bce45d, State: Running, Role: LEADER
I20260810 04:50:18.152595 20444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:18.152684 20458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb40312b04246f185c5527d72bce45d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33803 } }
I20260810 04:50:18.153051 20458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cb40312b04246f185c5527d72bce45d. Latest consensus state: current_term: 1 leader_uuid: "8cb40312b04246f185c5527d72bce45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb40312b04246f185c5527d72bce45d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33803 } } }
I20260810 04:50:18.153188 20458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:18.153416 20465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cb40312b04246f185c5527d72bce45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb40312b04246f185c5527d72bce45d" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33803 } } }
I20260810 04:50:18.153471 20477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:18.153554 20465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:18.154066 20477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:18.156160 20477 catalog_manager.cc:1347] Generated new cluster ID: d171ac13089c4084ad5f3f4020c2240d
I20260810 04:50:18.156258 20477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:18.156404 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:18.152715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38621 (local address 127.14.237.62:33803)
0810 04:50:18.152747 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:18.152748 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:18.152890 (+ 142us) server_negotiation.cc:407] Connection header received
0810 04:50:18.153000 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:18.153001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:18.153024 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:18.153074 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:18.153257 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:18.154023 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:18.155140 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:18.155406 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:18.155488 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:18.156121 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:18.156122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:18.156122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:18.156127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:18.156141 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:18.156145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:18.156221 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:18.156289 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 04:50:18.156309 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:50:18.174157 20477 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:18.174718 20477 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:18.182644 20477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cb40312b04246f185c5527d72bce45d: Generated new TSK 0
I20260810 04:50:18.182822 20477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:18.187173 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20343
2026-08-10T04:50:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:18 dist-test-slave-8hhm krb5kdc[20509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:18 dist-test-slave-8hhm krb5kdc[20509](info): set up 2 sockets
Aug 10 04:50:18 dist-test-slave-8hhm krb5kdc[20509](info): commencing operation
krb5kdc: starting...
W20260810 04:50:19.447292 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:19 dist-test-slave-8hhm krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:50:19.567322 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35923
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:35923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:19.647308 20529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:19.647492 20529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:19.647514 20529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:19.648639 20529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:19.648680 20529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:19.648694 20529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:19.648706 20529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:19.648741 20529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:19.650166 20529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:35923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:35923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:19.650394 20529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:19.650619 20529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:19.655659 20542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:19.655663 20536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:19.655771 20538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:19.655987 20529 server_base.cc:1034] running on GCE node
I20260810 04:50:19.656247 20529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:19.656546 20529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:19.657685 20529 hybrid_clock.cc:648] HybridClock initialized: now 1786337419657667 us; error 35 us; skew 500 ppm
Aug 10 04:50:19 dist-test-slave-8hhm krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337419, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:19.660243 20529 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:50:19.660719 20529 webserver.cc:460] Webserver started at http://127.14.237.62:42933/ using document root <none> and password file <none>
I20260810 04:50:19.660960 20529 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:19.661005 20529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:19.661167 20529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:19.662231 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "f7a28f70bbca4c08be7a4da8cf7509a3"
format_stamp: "Formatted at 2026-08-10 04:50:19 on dist-test-slave-8hhm"
server_key: "7fd98d3193f272dbbdfd346ad0eb6d81"
server_key_iv: "b0b183138ca141537bc2ab627c16a5a8"
server_key_version: "encryptionkey@0"
I20260810 04:50:19.662585 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "f7a28f70bbca4c08be7a4da8cf7509a3"
format_stamp: "Formatted at 2026-08-10 04:50:19 on dist-test-slave-8hhm"
server_key: "7fd98d3193f272dbbdfd346ad0eb6d81"
server_key_iv: "b0b183138ca141537bc2ab627c16a5a8"
server_key_version: "encryptionkey@0"
I20260810 04:50:19.663815 20529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:50:19.664593 20551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:19.664780 20529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:50:19.664884 20529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "f7a28f70bbca4c08be7a4da8cf7509a3"
format_stamp: "Formatted at 2026-08-10 04:50:19 on dist-test-slave-8hhm"
server_key: "7fd98d3193f272dbbdfd346ad0eb6d81"
server_key_iv: "b0b183138ca141537bc2ab627c16a5a8"
server_key_version: "encryptionkey@0"
I20260810 04:50:19.664968 20529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:19.668210 20529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:19.670090 20529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:19.670234 20529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:19.674283 20529 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:35923
I20260810 04:50:19.674316 20638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:35923 every 8 connection(s)
I20260810 04:50:19.674614 20529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:19.675581 20639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:19.677871 20639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3: Bootstrap starting.
I20260810 04:50:19.678390 20639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:19.678614 20639 log.cc:824] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:19.679267 20639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3: No bootstrap required, opened a new log
I20260810 04:50:19.680683 20639 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a28f70bbca4c08be7a4da8cf7509a3" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35923 } }
I20260810 04:50:19.680807 20639 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:50:19.680828 20639 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a28f70bbca4c08be7a4da8cf7509a3, State: Initialized, Role: FOLLOWER
I20260810 04:50:19.680918 20639 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a28f70bbca4c08be7a4da8cf7509a3" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35923 } }
I20260810 04:50:19.680958 20639 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:50:19.681001 20639 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:50:19.681046 20639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:19.681878 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20529
I20260810 04:50:19.681924 20639 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a28f70bbca4c08be7a4da8cf7509a3" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35923 } }
I20260810 04:50:19.682019 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:19.682042 20639 leader_election.cc:302] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7a28f70bbca4c08be7a4da8cf7509a3; no voters:
I20260810 04:50:19.682232 15284 external_mini_cluster.cc:1428] Setting key 55f3a71bb9d858f197d71e40fac147ab
I20260810 04:50:19.682272 20639 leader_election.cc:288] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:50:19.682292 20647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:19.682464 20647 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [term 1 LEADER]: Becoming Leader. State: Replica: f7a28f70bbca4c08be7a4da8cf7509a3, State: Running, Role: LEADER
I20260810 04:50:19.682593 20639 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:19.682608 20647 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a28f70bbca4c08be7a4da8cf7509a3" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35923 } }
I20260810 04:50:19.683048 20653 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7a28f70bbca4c08be7a4da8cf7509a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a28f70bbca4c08be7a4da8cf7509a3" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35923 } } }
I20260810 04:50:19.683230 20657 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7a28f70bbca4c08be7a4da8cf7509a3. Latest consensus state: current_term: 1 leader_uuid: "f7a28f70bbca4c08be7a4da8cf7509a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a28f70bbca4c08be7a4da8cf7509a3" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 35923 } } }
I20260810 04:50:19.683319 20653 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:19.683560 20657 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:19.684047 20672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:19.684665 20672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:19.685083 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:19.682853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34033 (local address 127.14.237.62:35923)
0810 04:50:19.683046 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:19.683056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:19.683242 (+ 186us) server_negotiation.cc:407] Connection header received
0810 04:50:19.683368 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:19.683375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:19.683408 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:19.683546 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:19.684615 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:19.684619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:19.684619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:19.684623 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:19.684683 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:19.684699 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:19.684783 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:19.684861 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 04:50:19.684906 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260810 04:50:19.686124 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:19.688067 20672 catalog_manager.cc:1347] Generated new cluster ID: 103f306d14a14004b45171540b43462f
I20260810 04:50:19.688122 20672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:19.690414 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:19.687731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53505 (local address 127.14.237.62:35923)
0810 04:50:19.687765 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:19.687767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:19.689464 (+ 1697us) server_negotiation.cc:407] Connection header received
0810 04:50:19.689539 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:19.689540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:19.689545 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:19.689608 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:19.690212 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:19.690213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:19.690213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:50:19.690217 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:19.690231 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:50:19.690235 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:19.690288 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:19.690338 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:50:19.690353 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:50:19.698832 20672 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:19.698993 20672 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:19.703162 20672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7a28f70bbca4c08be7a4da8cf7509a3: Generated new TSK 0
I20260810 04:50:19.703330 20672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:50:19.711930 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20529
2026-08-10T04:50:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1526 ms)
[----------] 18 tests from AuthTokenIssuingTest (27821 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:19 dist-test-slave-8hhm krb5kdc[20702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:19 dist-test-slave-8hhm krb5kdc[20702](info): set up 2 sockets
Aug 10 04:50:19 dist-test-slave-8hhm krb5kdc[20702](info): commencing operation
krb5kdc: starting...
W20260810 04:50:20.977458 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:50:21.099475 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:33353
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:33353,127.14.237.61:43307,127.14.237.60:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:21.181078 20720 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:50:21.181304 20720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:21.181322 20720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:21.181340 20720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:21.182473 20720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:21.182514 20720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:21.182528 20720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:21.182540 20720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:21.182596 20720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:21.184064 20720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:33353,127.14.237.61:43307,127.14.237.60:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:33353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:21.184332 20720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:21.184567 20720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:21.189584 20730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:21.189558 20735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:21.189569 20729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:21.189764 20720 server_base.cc:1034] running on GCE node
I20260810 04:50:21.190124 20720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:21.190405 20720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:21.191593 20720 hybrid_clock.cc:648] HybridClock initialized: now 1786337421191578 us; error 24 us; skew 500 ppm
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:21.194161 20720 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:50:21.194626 20720 webserver.cc:460] Webserver started at http://127.14.237.62:46245/ using document root <none> and password file <none>
I20260810 04:50:21.194869 20720 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:21.194938 20720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:21.195086 20720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:21.196151 20720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "e30504bce6ee4160887727c28bdd57b9"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "668995a77d59ee0ed15456572bf24d3a"
server_key_iv: "794d5ee39f885186c6970b9bc5bf30bb"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.196537 20720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "e30504bce6ee4160887727c28bdd57b9"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "668995a77d59ee0ed15456572bf24d3a"
server_key_iv: "794d5ee39f885186c6970b9bc5bf30bb"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.198004 20720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:50:21.198784 20745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:21.198968 20720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:50:21.199074 20720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "e30504bce6ee4160887727c28bdd57b9"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "668995a77d59ee0ed15456572bf24d3a"
server_key_iv: "794d5ee39f885186c6970b9bc5bf30bb"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.199150 20720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:21.207737 20720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:21.209524 20720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:21.209679 20720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:21.213742 20720 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:33353
I20260810 04:50:21.213789 20829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:33353 every 8 connection(s)
I20260810 04:50:21.214130 20720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:21.214439 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20720
I20260810 04:50:21.214592 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:21.214781 15284 external_mini_cluster.cc:1428] Setting key 4ca3bf8d5773c424fb7e7c7d01d86710
I20260810 04:50:21.215232 20830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:21.217000 20830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.237.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.61@KRBTEST.COM" created.
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:50:21.231624 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.217833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:33353 (local address 127.0.0.1:41706)
0810 04:50:21.218087 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.218132 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.218875 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.219240 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.219244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.219618 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.219859 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.219867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.220976 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.220978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.222138 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.222142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.222645 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.222646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.222736 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.223233 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.223244 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.227726 (+ 4482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.230133 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.230140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.230142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.230336 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.230624 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.230624 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.230625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.230705 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.231196 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.231202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.231267 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.231341 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.231466 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":128,"threads_started":1}
I20260810 04:50:21.231710 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.217914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.14.237.62:33353)
0810 04:50:21.218159 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.218163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.218236 (+ 73us) server_negotiation.cc:407] Connection header received
0810 04:50:21.219067 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.219069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.219085 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.219190 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.219984 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.220865 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.222230 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.222526 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.222614 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.227826 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.227835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.227835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.227842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.230050 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.230424 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.230427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.230428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.230548 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.230776 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.230778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.230778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.231116 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.231223 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.231413 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.231558 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260810 04:50:21.232482 20830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:21.232826 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.232703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:59208)
0810 04:50:21.232756 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.232803 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 04:50:21.232960 20751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
I20260810 04:50:21.233389 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.233296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:59220)
0810 04:50:21.233345 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.233368 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 04:50:21.233495 20830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:43307: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab.
Entry for principal kudu/127.14.237.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab.
I20260810 04:50:21.242444 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.61:43307
--webserver_interface=127.14.237.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab
--principal=kudu/127.14.237.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:33353,127.14.237.61:43307,127.14.237.60:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:50:21.282725 20830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } attempt: 1
I20260810 04:50:21.283318 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.283172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:59222)
0810 04:50:21.283241 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.283288 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 04:50:21.283710 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.283624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:59224)
0810 04:50:21.283665 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.283691 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 04:50:21.283824 20830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:43307: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
W20260810 04:50:21.324779 20850 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:50:21.325031 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:21.325062 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:21.325130 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:21.326318 20850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:21.326391 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:21.326416 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:21.326447 20850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:21.326475 20850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:21.327960 20850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:33353,127.14.237.61:43307,127.14.237.60:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab
--principal=kudu/127.14.237.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.61:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.237.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:21.328249 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:21.328478 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:21.333706 20857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:21.333679 20858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:21.333730 20866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:21.333915 20850 server_base.cc:1034] running on GCE node
I20260810 04:50:21.334127 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:21.334355 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:21.335533 20850 hybrid_clock.cc:648] HybridClock initialized: now 1786337421335523 us; error 27 us; skew 500 ppm
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:21.338155 20850 init.cc:377] Logged in from keytab as kudu/127.14.237.61@KRBTEST.COM (short username kudu)
I20260810 04:50:21.338579 20850 webserver.cc:460] Webserver started at http://127.14.237.61:43585/ using document root <none> and password file <none>
I20260810 04:50:21.338800 20850 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:21.338845 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:21.338985 20850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:21.340021 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/data/instance:
uuid: "f97a5a6fb14d4d55b542622126641000"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "e156690bdcc801fb8444d9e48d7a49d5"
server_key_iv: "0a634ef94d0a1e8530e8689e17420e40"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.340387 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/wal/instance:
uuid: "f97a5a6fb14d4d55b542622126641000"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "e156690bdcc801fb8444d9e48d7a49d5"
server_key_iv: "0a634ef94d0a1e8530e8689e17420e40"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.341820 20850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:50:21.342664 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:21.342875 20850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:50:21.342974 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/wal
uuid: "f97a5a6fb14d4d55b542622126641000"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "e156690bdcc801fb8444d9e48d7a49d5"
server_key_iv: "0a634ef94d0a1e8530e8689e17420e40"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.343051 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:21.351976 20830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } attempt: 2
I20260810 04:50:21.352396 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.352265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:53670)
0810 04:50:21.352327 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.352365 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:50:21.352746 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.352645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:53678)
0810 04:50:21.352693 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.352718 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:50:21.352829 20830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:43307: Network error: Client connection negotiation failed: client connection to 127.14.237.61:43307: connect: Connection refused (error 111)
I20260810 04:50:21.367846 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:21.369824 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:21.369993 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:21.374133 20850 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.61:43307
I20260810 04:50:21.374181 20966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.61:43307 every 8 connection(s)
I20260810 04:50:21.374565 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
I20260810 04:50:21.375605 20967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:21.377269 20967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:21.377594 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20850
I20260810 04:50:21.377722 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-1/wal/instance
I20260810 04:50:21.377925 15284 external_mini_cluster.cc:1428] Setting key cb7c4321f6e22bd1ae6ef3cea75063ff
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:50:21.387624 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.378133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.14.237.62:33353)
0810 04:50:21.378187 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.378189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.378453 (+ 264us) server_negotiation.cc:407] Connection header received
0810 04:50:21.378956 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.378958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.378973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.379062 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.379768 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.380265 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.381787 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.381973 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.382021 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.384440 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.384442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.384442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.384451 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.386236 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.386673 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.386675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.386675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.386743 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.386980 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.386981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.386981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.387126 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.387207 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.387448 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.387535 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.14.237.60@KRBTEST.COM; defaulting to no policy
I20260810 04:50:21.387768 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.378127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:33353 (local address 127.0.0.1:33546)
0810 04:50:21.378315 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.378357 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.378850 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.379136 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.379140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.379387 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.379632 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.379638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.380367 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.380368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.381692 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.381695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.382069 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.382070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.382233 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.382737 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.382757 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.384342 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.386340 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.386347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.386348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.386565 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.386811 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.386811 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.386812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.386908 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.387200 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.387205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.387320 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.387399 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.387509 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260810 04:50:21.388419 20967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.60@KRBTEST.COM" created.
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:50:21.396648 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.388696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:53692)
0810 04:50:21.388750 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.388761 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.388819 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.389328 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.389329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.389439 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.389486 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.389487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.390363 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.390364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.391036 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.391039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.391488 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.391489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.391569 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.391995 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.392001 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.393425 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.395672 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.395672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.395673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.395834 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.396068 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.396068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.396069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.396119 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.396443 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.396444 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.396476 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.396525 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.396589 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:50:21.396783 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.388934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.14.237.61:43307)
0810 04:50:21.389104 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.389108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.389123 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:50:21.389142 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.389143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.389154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.389326 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.389538 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.390273 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.391129 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.391412 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.391469 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.393516 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.393525 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.393525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.393532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.395560 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.395922 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.395926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.395927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.395995 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.396177 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.396178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.396179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.396369 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.396483 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.396567 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.396664 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260810 04:50:21.397397 20967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:21.397749 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.397648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44740)
0810 04:50:21.397689 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.397730 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 04:50:21.397917 20881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
I20260810 04:50:21.398281 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.398219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44756)
0810 04:50:21.398243 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.398265 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260810 04:50:21.398415 20967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:40931: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab.
Entry for principal kudu/127.14.237.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab.
I20260810 04:50:21.407696 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.60:40931
--webserver_interface=127.14.237.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab
--principal=kudu/127.14.237.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:33353,127.14.237.61:43307,127.14.237.60:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:50:21.443984 20830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } attempt: 3
I20260810 04:50:21.447639 20967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } attempt: 1
I20260810 04:50:21.448041 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.447932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44762)
0810 04:50:21.447983 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.448014 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:50:21.448428 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.448342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44770)
0810 04:50:21.448377 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.448398 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 04:50:21.448518 20967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:40931: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
I20260810 04:50:21.451167 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.444454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.14.237.61:43307)
0810 04:50:21.444527 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.444529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.444540 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:50:21.444571 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.444572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.444585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.444657 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.444985 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.445577 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.446301 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.446515 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.446599 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.448696 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.448697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.448697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.448703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.450096 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.450496 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.450497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.450497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.450548 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.450744 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.450744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.450745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.450837 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.450886 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.451061 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.451115 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:50:21.451238 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.444354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:53694)
0810 04:50:21.444422 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.444441 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.444512 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.444666 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.444667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.444834 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.444899 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.444901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.445664 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.445665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.446209 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.446211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.446606 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.446608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.446705 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.447204 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.447208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.448613 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.450181 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.450182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.450182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.450419 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.450612 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.450613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.450614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.450680 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.450903 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.450904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.450961 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.451047 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.451146 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 04:50:21.451635 20830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:21.451990 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.451871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44784)
0810 04:50:21.451929 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.451964 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 04:50:21.452322 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.452232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44786)
0810 04:50:21.452276 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.452302 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:50:21.452418 20830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:40931: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
I20260810 04:50:21.483577 20830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } attempt: 1
I20260810 04:50:21.484051 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.483909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44792)
0810 04:50:21.483973 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.484016 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 04:50:21.484450 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.484330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44804)
0810 04:50:21.484402 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.484428 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 04:50:21.484591 20830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:40931: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
W20260810 04:50:21.490644 20978 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:50:21.490872 20978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:21.490908 20978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:21.490969 20978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:21.492094 20978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:21.492166 20978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:21.492193 20978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:21.492220 20978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:21.492249 20978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:21.493800 20978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:33353,127.14.237.61:43307,127.14.237.60:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab
--principal=kudu/127.14.237.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.60:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.237.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:21.494072 20978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:21.494288 20978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:21.499032 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:21.499150 20983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:21.499055 20996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:21.499460 20978 server_base.cc:1034] running on GCE node
I20260810 04:50:21.499761 20978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:21.500041 20978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:21.501222 20978 hybrid_clock.cc:648] HybridClock initialized: now 1786337421501192 us; error 31 us; skew 500 ppm
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:21.503800 20978 init.cc:377] Logged in from keytab as kudu/127.14.237.60@KRBTEST.COM (short username kudu)
I20260810 04:50:21.504158 20978 webserver.cc:460] Webserver started at http://127.14.237.60:41871/ using document root <none> and password file <none>
I20260810 04:50:21.504324 20978 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:21.504362 20978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:21.504446 20978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:21.505492 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/data/instance:
uuid: "f9221ec8eb584b1aac5eead300379e9f"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "f16f84e3bf2b6b726f7f7fce2854317d"
server_key_iv: "240d8a32a8209f0471957ab230d41910"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.505801 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/wal/instance:
uuid: "f9221ec8eb584b1aac5eead300379e9f"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "f16f84e3bf2b6b726f7f7fce2854317d"
server_key_iv: "240d8a32a8209f0471957ab230d41910"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.507014 20978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:50:21.507665 21010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:21.507838 20978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:50:21.507917 20978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/wal
uuid: "f9221ec8eb584b1aac5eead300379e9f"
format_stamp: "Formatted at 2026-08-10 04:50:21 on dist-test-slave-8hhm"
server_key: "f16f84e3bf2b6b726f7f7fce2854317d"
server_key_iv: "240d8a32a8209f0471957ab230d41910"
server_key_version: "encryptionkey@0"
I20260810 04:50:21.507968 20978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:21.516647 20967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } attempt: 2
I20260810 04:50:21.517027 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.516911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44816)
0810 04:50:21.516967 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.517001 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:50:21.517392 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.517300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44822)
0810 04:50:21.517351 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.517374 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 04:50:21.517504 20967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:40931: Network error: Client connection negotiation failed: client connection to 127.14.237.60:40931: connect: Connection refused (error 111)
I20260810 04:50:21.524356 20978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:21.526445 20978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:21.526602 20978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:21.530552 20978 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.60:40931
I20260810 04:50:21.530616 21099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.60:40931 every 8 connection(s)
I20260810 04:50:21.530963 20978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
I20260810 04:50:21.531940 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:21.532491 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 20978
I20260810 04:50:21.532573 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786337145367972-15284-0/minicluster-data/master-2/wal/instance
I20260810 04:50:21.532763 15284 external_mini_cluster.cc:1428] Setting key db45aec995014158455555e4027e1b57
I20260810 04:50:21.533901 21100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
I20260810 04:50:21.543660 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.534596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.14.237.62:33353)
0810 04:50:21.534672 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.534674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.534811 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:50:21.534882 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.534883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.534894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.534958 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.535391 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.536107 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.537341 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.537620 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.537684 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.540052 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.540058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.540059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.540066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.541874 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.542329 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.542330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.542331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.542397 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.543013 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.543014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.543014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.543171 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.543246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.543461 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.543549 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 04:50:21.543944 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.533289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.14.237.60:40931)
0810 04:50:21.533445 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.533453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.533512 (+ 59us) server_negotiation.cc:407] Connection header received
0810 04:50:21.533913 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.533919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.533960 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.534064 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.534835 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.536053 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.537150 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.537601 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.537707 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.540604 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.540612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.540613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.540620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.542400 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.542707 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.542709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.542709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.542871 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.543074 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.543074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.543075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.543334 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.543465 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.543546 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.543651 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260810 04:50:21.544124 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.534534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:33353 (local address 127.0.0.1:33556)
0810 04:50:21.534706 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.534748 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.534814 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.534983 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.534985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.535223 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.535302 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.535306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.536193 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.536194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.537275 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.537278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.537721 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.537722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.537784 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.538359 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.538369 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.539951 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.541963 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.541969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.541971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.542251 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.542727 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.542728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.542729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.542934 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.543244 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.543249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.543317 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.543420 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.543589 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260810 04:50:21.544624 21100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:50:21.551334 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.544845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:43307 (local address 127.0.0.1:53708)
0810 04:50:21.544977 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.544985 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.545016 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.545136 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.545136 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.545302 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.545350 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.545351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.546104 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.546105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.546574 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.546575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.546980 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.546981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.547052 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.547413 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.547417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.548616 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.550252 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.550253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.550253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.550466 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.550705 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.550706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.550706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.550769 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.551037 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.551038 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.551077 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.551152 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.551248 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260810 04:50:21.551343 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.544919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.14.237.61:43307)
0810 04:50:21.544962 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.544964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.545011 (+ 47us) server_negotiation.cc:407] Connection header received
0810 04:50:21.545057 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.545058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.545068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.545134 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.545424 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.546039 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.546671 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.546905 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.546968 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.548698 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.548699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.548700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.548707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.550188 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.550565 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.550566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.550567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.550636 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.550836 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.550837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.550837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.550950 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.551033 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.551190 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.551262 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:50:21.551641 21100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:21.552160 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.544709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.14.237.62:33353)
0810 04:50:21.544769 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.544771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.544803 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:50:21.544881 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.544882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.544890 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.544941 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.545559 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.546064 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.546749 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.547002 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.547065 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.549489 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.549490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.549491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.549498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.550869 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.551219 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.551220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.551220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.551284 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.551496 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.551497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.551497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.551621 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.551820 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.551891 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.551973 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:50:21.558511 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.551921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44834)
0810 04:50:21.551982 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.551991 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.552025 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.552340 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.552341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.552494 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.552520 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.552522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.553364 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.553365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.553841 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.553843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.554239 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.554240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.554291 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.554599 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.554604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.555837 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.557519 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.557520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.557521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.557698 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.557906 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.557906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.557906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.557960 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.558184 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.558185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.558228 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.558311 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.558404 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 04:50:21.558573 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.551955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.14.237.60:40931)
0810 04:50:21.552238 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.552240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.552246 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:50:21.552261 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.552261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.552267 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.552338 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.552621 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.553286 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.553945 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.554154 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.554216 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.555920 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.555921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.555921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.555928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.557449 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.557778 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.557779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.557779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.557828 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.558019 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.558020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.558020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.558120 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.558192 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.558333 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.558455 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":269}
I20260810 04:50:21.559768 20830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } attempt: 2
I20260810 04:50:21.560050 21100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f: Bootstrap starting.
I20260810 04:50:21.560223 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.552796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.14.237.61:43307)
0810 04:50:21.552831 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.552833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.552841 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:50:21.552887 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.552888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.552896 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.552958 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.553484 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.553849 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.554688 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.554870 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.554919 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.557372 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.557373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.557374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.557380 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.558967 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.559319 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.559321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.559321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.559395 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.559589 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.559590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.559590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.559696 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.560017 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.560063 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.560143 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:50:21.561106 21100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:21.561493 21100 log.cc:824] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:21.561839 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:50:21.562353 21100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f: No bootstrap required, opened a new log
I20260810 04:50:21.562879 20789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e30504bce6ee4160887727c28bdd57b9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33572
I20260810 04:50:21.564007 21100 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } }
I20260810 04:50:21.564160 21100 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9221ec8eb584b1aac5eead300379e9f, State: Initialized, Role: FOLLOWER
I20260810 04:50:21.564260 21100 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } }
I20260810 04:50:21.564621 21100 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:21.564594 21116 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.564689 21116 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
W20260810 04:50:21.566820 21137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:50:21.566973 21137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:50:21.568761 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.559993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44836)
0810 04:50:21.560066 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.560076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.560113 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.560395 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.560396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.560559 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.560588 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.560590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.561409 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.561410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.562760 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.562763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.563899 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.563900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.563955 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.564347 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.564352 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.565720 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.567556 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.567557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.567557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.567806 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.568042 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.568043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.568043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.568127 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.568449 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.568450 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.568494 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.568572 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.568661 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 04:50:21.568769 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.560189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.14.237.60:40931)
0810 04:50:21.560249 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.560251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.560259 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:50:21.560279 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.560280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.560290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.560650 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.560673 (+ 23us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.561338 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.563625 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.563819 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.563900 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.565808 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.565809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.565809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.565817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.567464 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.567897 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.567898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.567899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.567961 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.568204 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.568205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.568206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.568368 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.568441 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.568618 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.568690 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:50:21.566819 20978 master_runner.cc:186] Error getting master registration for 127.14.237.62:33353: OK, instance_id { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" instance_seqno: 1786337421212572 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:50:21.569093 20978 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:50:21.570258 20830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9: Bootstrap starting.
I20260810 04:50:21.570955 20830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:21.571233 20830 log.cc:824] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:21.572042 20830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9: No bootstrap required, opened a new log
I20260810 04:50:21.573007 20830 raft_consensus.cc:348] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } }
I20260810 04:50:21.573146 20830 raft_consensus.cc:729] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30504bce6ee4160887727c28bdd57b9, State: Initialized, Role: FOLLOWER
I20260810 04:50:21.573264 20830 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } }
I20260810 04:50:21.573539 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.573592 20830 sys_catalog.cc:564] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:21.573634 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:21.573752 20789 raft_consensus.cc:482] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 04:50:21.573819 20789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:21.574656 20789 raft_consensus.cc:504] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } }
I20260810 04:50:21.574900 20789 leader_election.cc:288] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [CANDIDATE]: Term 1 election: Requested vote from peers f97a5a6fb14d4d55b542622126641000 (127.14.237.61:43307), f9221ec8eb584b1aac5eead300379e9f (127.14.237.60:40931)
I20260810 04:50:21.575330 21054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e30504bce6ee4160887727c28bdd57b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f9221ec8eb584b1aac5eead300379e9f"
I20260810 04:50:21.575433 21054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:21.576323 21054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30504bce6ee4160887727c28bdd57b9 in term 1.
I20260810 04:50:21.576541 20751 leader_election.cc:302] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e30504bce6ee4160887727c28bdd57b9, f9221ec8eb584b1aac5eead300379e9f; no voters:
I20260810 04:50:21.576701 21143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:21.576874 21143 raft_consensus.cc:686] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [term 1 LEADER]: Becoming Leader. State: Replica: e30504bce6ee4160887727c28bdd57b9, State: Running, Role: LEADER
I20260810 04:50:21.576960 21143 consensus_queue.cc:237] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } }
I20260810 04:50:21.577390 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30504bce6ee4160887727c28bdd57b9. Latest consensus state: current_term: 1 leader_uuid: "e30504bce6ee4160887727c28bdd57b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.577467 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:21.577947 20720 master_runner.cc:186] Error getting master registration for 127.14.237.61:43307: OK, instance_id { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" instance_seqno: 1786337421372955 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:50:21.580711 20720 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 04:50:21.581115 21166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:50:21.581164 21166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:50:21.581321 21181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:21.581465 21181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:21.586375 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.576605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.14.237.60:40931)
0810 04:50:21.576639 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.576640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.577479 (+ 839us) server_negotiation.cc:407] Connection header received
0810 04:50:21.577571 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.577571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.577581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.577622 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.578351 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.579080 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.580048 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.580319 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.580354 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.581937 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.581939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.581939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.581946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.583668 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.585496 (+ 1828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.585498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.585499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.585556 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.585783 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.585784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.585784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.585911 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.585992 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.586201 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.586279 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260810 04:50:21.587422 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.576078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.14.237.61:43307)
0810 04:50:21.576127 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.576128 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.577281 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 04:50:21.577381 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.577382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.577393 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.577443 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.578652 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.579319 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.580616 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.580926 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.580970 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.583499 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.583500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.583500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.583519 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.585314 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.585928 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.585929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.585929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.585997 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.586710 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.586711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.586712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.586858 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.586945 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.587256 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.587327 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:50:21.590051 21054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [term 1 FOLLOWER]: Refusing update from remote peer e30504bce6ee4160887727c28bdd57b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:50:21.590314 21143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:50:21.591432 21116 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30504bce6ee4160887727c28bdd57b9. Latest consensus state: current_term: 1 leader_uuid: "e30504bce6ee4160887727c28bdd57b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.591516 21116 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:21.593187 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.576851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.14.237.62:33353)
0810 04:50:21.576886 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.576888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.584657 (+ 7769us) server_negotiation.cc:407] Connection header received
0810 04:50:21.584775 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.584776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.584786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.584819 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.586689 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.587378 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.588317 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.588627 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.588671 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.590221 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.590222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.590222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.590245 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.591856 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.592282 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.592284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.592284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.592358 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.592595 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.592596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.592596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.592721 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.592817 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.592988 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.593073 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:50:21.594532 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e30504bce6ee4160887727c28bdd57b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.594615 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:21.595237 21116 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e30504bce6ee4160887727c28bdd57b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.595320 21116 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:21.595422 21185 mvcc.cc:205] Tried to move back new op lower bound from 7316838078830497792 to 7316838078779944960. Current Snapshot: MvccSnapshot[applied={T|T < 7316838078830497792}]
I20260810 04:50:21.595665 21184 mvcc.cc:205] Tried to move back new op lower bound from 7316838078830497792 to 7316838078779944960. Current Snapshot: MvccSnapshot[applied={T|T < 7316838078830497792}]
I20260810 04:50:21.596148 21181 catalog_manager.cc:1347] Generated new cluster ID: 69032af991e54081a9fbdaa2a6750803
I20260810 04:50:21.596200 21181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:50:21.608665 20967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } attempt: 3
Aug 10 04:50:21 dist-test-slave-8hhm krb5kdc[20702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337421, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
I20260810 04:50:21.617274 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.608892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:40931 (local address 127.0.0.1:44862)
0810 04:50:21.608935 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:21.608946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:21.609002 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:21.609153 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:21.609154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:21.609340 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:21.609379 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.609381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.610278 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.610279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.611065 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:21.611067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.611627 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.611628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:21.611688 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.612177 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:21.612182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:21.613649 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:21.616131 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.616132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.616132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.616335 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.616557 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:21.616558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:21.616558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:21.616629 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:21.616918 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:21.616919 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:21.616987 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 04:50:21.617070 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:50:21.617168 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 04:50:21.617791 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:21.608956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.14.237.60:40931)
0810 04:50:21.609008 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.609010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.609034 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:50:21.609054 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.609055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.609063 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.609113 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.609458 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.610180 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.611192 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:21.611491 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:21.611543 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:21.613744 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:21.613745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:21.613746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:21.613752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:21.615396 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.616421 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.616422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.616422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.616487 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:21.616700 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:21.616701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:21.616701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:21.616823 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:21.616883 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:21.617645 (+ 762us) server_negotiation.cc:299] Negotiation successful
0810 04:50:21.617713 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:50:21.619472 20967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000: Bootstrap starting.
I20260810 04:50:21.620172 20967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:21.620481 20967 log.cc:824] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:21.626237 20967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000: No bootstrap required, opened a new log
I20260810 04:50:21.628504 20967 raft_consensus.cc:348] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } }
I20260810 04:50:21.628644 20967 raft_consensus.cc:729] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f97a5a6fb14d4d55b542622126641000, State: Initialized, Role: FOLLOWER
I20260810 04:50:21.628728 20967 consensus_queue.cc:260] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } }
I20260810 04:50:21.629124 20925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:21.629280 21181 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:21.629947 21181 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:21.630069 20925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [term 1 FOLLOWER]: Refusing update from remote peer e30504bce6ee4160887727c28bdd57b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:50:21.630334 21190 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:50:21.629159 20926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e30504bce6ee4160887727c28bdd57b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f97a5a6fb14d4d55b542622126641000"
I20260810 04:50:21.629066 20967 sys_catalog.cc:564] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:21.629033 21194 sys_catalog.cc:455] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.633051 21194 sys_catalog.cc:458] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:21.633153 21194 sys_catalog.cc:455] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30504bce6ee4160887727c28bdd57b9. Latest consensus state: current_term: 1 leader_uuid: "e30504bce6ee4160887727c28bdd57b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.633286 21194 sys_catalog.cc:458] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:21.633349 21194 sys_catalog.cc:455] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e30504bce6ee4160887727c28bdd57b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.633428 21194 sys_catalog.cc:458] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:21.637795 20926 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e30504bce6ee4160887727c28bdd57b9 for term 1 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 04:50:21.638363 21198 mvcc.cc:205] Tried to move back new op lower bound from 7316838078978568192 to 7316838078779944960. Current Snapshot: MvccSnapshot[applied={T|T < 7316838078830497792}]
W20260810 04:50:21.638921 21221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:50:21.639014 21221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:50:21.639806 21190 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e30504bce6ee4160887727c28bdd57b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30504bce6ee4160887727c28bdd57b9" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 33353 } } peers { permanent_uuid: "f97a5a6fb14d4d55b542622126641000" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 43307 } } peers { permanent_uuid: "f9221ec8eb584b1aac5eead300379e9f" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 40931 } } }
I20260810 04:50:21.639885 21190 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9 [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:21.641526 21181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e30504bce6ee4160887727c28bdd57b9: Generated new TSK 0
I20260810 04:50:21.641862 21181 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:50:21.678956 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.677531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.14.237.61:43307)
0810 04:50:21.677572 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.677574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.678348 (+ 774us) server_negotiation.cc:407] Connection header received
0810 04:50:21.678450 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.678451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.678461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.678489 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.678893 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53726: BlockingRecv error: recv got EOF from 127.0.0.1:53726 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 04:50:21.679682 20840 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.677726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.14.237.62:33353)
0810 04:50:21.677760 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.677761 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.679164 (+ 1403us) server_negotiation.cc:407] Connection header received
0810 04:50:21.679279 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.679280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.679290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.679317 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.679622 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33590: BlockingRecv error: recv got EOF from 127.0.0.1:33590 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:50:21.680434 21108 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.678039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.14.237.60:40931)
0810 04:50:21.678073 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.678075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.679914 (+ 1839us) server_negotiation.cc:407] Connection header received
0810 04:50:21.680056 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.680057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.680066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.680095 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.680379 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44870: BlockingRecv error: recv got EOF from 127.0.0.1:44870 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:50:21.682717 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.681940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.14.237.61:43307)
0810 04:50:21.681969 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.681970 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.682172 (+ 202us) server_negotiation.cc:407] Connection header received
0810 04:50:21.682307 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.682307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.682323 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.682351 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.682673 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53742: BlockingRecv error: recv got EOF from 127.0.0.1:53742 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260810 04:50:21.685616 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.684861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.14.237.61:43307)
0810 04:50:21.684895 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.684896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.685089 (+ 193us) server_negotiation.cc:407] Connection header received
0810 04:50:21.685178 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.685179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.685188 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.685216 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.685575 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53754: BlockingRecv error: recv got EOF from 127.0.0.1:53754 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:50:21.689471 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.688719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.14.237.61:43307)
0810 04:50:21.688749 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.688750 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.688932 (+ 182us) server_negotiation.cc:407] Connection header received
0810 04:50:21.689036 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.689036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.689045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.689073 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.689431 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53762: BlockingRecv error: recv got EOF from 127.0.0.1:53762 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 04:50:21.695104 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.694445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.14.237.61:43307)
0810 04:50:21.694473 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.694475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.694694 (+ 219us) server_negotiation.cc:407] Connection header received
0810 04:50:21.694776 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.694777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.694785 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.694811 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.695069 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53776: BlockingRecv error: recv got EOF from 127.0.0.1:53776 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 04:50:21.704828 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.704114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.14.237.61:43307)
0810 04:50:21.704145 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.704147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.704368 (+ 221us) server_negotiation.cc:407] Connection header received
0810 04:50:21.704458 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.704458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.704468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.704495 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.704785 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53780: BlockingRecv error: recv got EOF from 127.0.0.1:53780 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 04:50:21.722889 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.722050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.14.237.61:43307)
0810 04:50:21.722086 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.722088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.722349 (+ 261us) server_negotiation.cc:407] Connection header received
0810 04:50:21.722453 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.722454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.722466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.722495 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.722842 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53784: BlockingRecv error: recv got EOF from 127.0.0.1:53784 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 04:50:21.757313 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.756322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.14.237.61:43307)
0810 04:50:21.756368 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.756370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.756647 (+ 277us) server_negotiation.cc:407] Connection header received
0810 04:50:21.756765 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.756766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.756779 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.756810 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.757205 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53790: BlockingRecv error: recv got EOF from 127.0.0.1:53790 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:50:21.823552 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.822804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.14.237.61:43307)
0810 04:50:21.822874 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.822876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.822979 (+ 103us) server_negotiation.cc:407] Connection header received
0810 04:50:21.823039 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.823040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.823051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.823106 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.823484 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53806: BlockingRecv error: recv got EOF from 127.0.0.1:53806 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 04:50:21.955842 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:21.952783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.14.237.61:43307)
0810 04:50:21.952860 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:21.952862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:21.955271 (+ 2409us) server_negotiation.cc:407] Connection header received
0810 04:50:21.955334 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:21.955335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:21.955347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:21.955389 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:21.955778 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53812: BlockingRecv error: recv got EOF from 127.0.0.1:53812 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 04:50:22.213774 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:22.213027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.14.237.61:43307)
0810 04:50:22.213098 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:22.213099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:22.213140 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:50:22.213226 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:22.213227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:22.213236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:22.213300 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:22.213705 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53828: BlockingRecv error: recv got EOF from 127.0.0.1:53828 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:50:22.567441 21137 catalog_manager.cc:1259] Loaded cluster ID: 69032af991e54081a9fbdaa2a6750803
I20260810 04:50:22.567533 21137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f: loading cluster ID for follower catalog manager: success
I20260810 04:50:22.568305 21137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f: acquiring CA information for follower catalog manager: success
I20260810 04:50:22.568619 21137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9221ec8eb584b1aac5eead300379e9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:50:22.639535 21221 catalog_manager.cc:1259] Loaded cluster ID: 69032af991e54081a9fbdaa2a6750803
I20260810 04:50:22.639647 21221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000: loading cluster ID for follower catalog manager: success
I20260810 04:50:22.640538 21221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000: acquiring CA information for follower catalog manager: success
I20260810 04:50:22.640887 21221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f97a5a6fb14d4d55b542622126641000: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:50:22.729889 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:22.726993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.14.237.61:43307)
0810 04:50:22.727135 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:22.727138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:22.727152 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:50:22.727233 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:22.727234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:22.727314 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:22.727425 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:22.727612 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.728204 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.729000 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.729205 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.729349 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:22.729528 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:22.729649 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:22.729751 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:22.729770 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 04:50:22.729791 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":55,"threads_started":1}
I20260810 04:50:22.734771 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:22.732077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.14.237.60:40931)
0810 04:50:22.732199 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:22.732201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:22.732211 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:50:22.732270 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:22.732271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:22.732304 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:22.732383 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:22.732511 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.733030 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.733851 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.734048 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.734102 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:22.734362 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:22.734493 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:22.734579 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:22.734633 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:50:22.734656 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260810 04:50:22.739382 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:22.736350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.14.237.61:43307)
0810 04:50:22.736435 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:22.736437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:22.736572 (+ 135us) server_negotiation.cc:407] Connection header received
0810 04:50:22.736676 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:22.736677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:22.736687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:22.736746 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:22.736930 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.737491 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.738585 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.738887 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.738955 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:22.739097 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:22.739158 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:22.739210 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:22.739292 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 04:50:22.739313 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 04:50:22.739804 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:22.736343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.14.237.62:33353)
0810 04:50:22.736683 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:22.736686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:22.736696 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:50:22.736742 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:22.736742 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:22.736808 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:22.736883 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:22.737071 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.737670 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.738714 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.739057 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.739113 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:22.739353 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:22.739460 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:22.739528 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:22.739614 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:50:22.739640 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":51,"threads_started":1}
I20260810 04:50:22.740227 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:22.736680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.14.237.60:40931)
0810 04:50:22.736725 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:22.736727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:22.736823 (+ 96us) server_negotiation.cc:407] Connection header received
0810 04:50:22.736869 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:22.736870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:22.736879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:22.736932 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:22.737711 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.738159 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.739170 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:22.739452 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:22.739493 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:22.739881 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:22.739919 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:22.739948 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:22.740112 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 04:50:22.740136 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:50:22.741544 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20720
I20260810 04:50:22.763950 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20850
I20260810 04:50:22.778780 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 20978
2026-08-10T04:50:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3066 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:50:22 dist-test-slave-8hhm krb5kdc[21368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:50:22 dist-test-slave-8hhm krb5kdc[21368](info): set up 2 sockets
Aug 10 04:50:22 dist-test-slave-8hhm krb5kdc[21368](info): commencing operation
krb5kdc: starting...
W20260810 04:50:24.072510 15284 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/test-user.keytab.
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.237.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
Entry for principal kudu/127.14.237.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab.
I20260810 04:50:24.192401 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:32829
--webserver_interface=127.14.237.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:32829,127.14.237.61:44059,127.14.237.60:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:50:24.273030 21392 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:50:24.273265 21392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:24.273284 21392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:24.273341 21392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:24.274852 21392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:24.274894 21392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:24.274909 21392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:24.274946 21392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:24.274972 21392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:24.276448 21392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:32829,127.14.237.61:44059,127.14.237.60:41731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.62.keytab
--principal=kudu/127.14.237.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.62:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.237.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:24.276698 21392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:24.276947 21392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:24.281672 21409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:24.281797 21392 server_base.cc:1034] running on GCE node
W20260810 04:50:24.281683 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:24.281692 21401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:24.282111 21392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:24.282418 21392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:24.283587 21392 hybrid_clock.cc:648] HybridClock initialized: now 1786337424283560 us; error 38 us; skew 500 ppm
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:24.286111 21392 init.cc:377] Logged in from keytab as kudu/127.14.237.62@KRBTEST.COM (short username kudu)
I20260810 04:50:24.286582 21392 webserver.cc:460] Webserver started at http://127.14.237.62:35645/ using document root <none> and password file <none>
I20260810 04:50:24.286835 21392 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:24.286881 21392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:24.287061 21392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:24.288010 21392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/data/instance:
uuid: "0c88ae23cfa7463c9d83448ef6bf687a"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "4e453abb8e4dda2c8de2081d1097c0e1"
server_key_iv: "1d20331b99e69b3709ddd076cff5debd"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.288353 21392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/wal/instance:
uuid: "0c88ae23cfa7463c9d83448ef6bf687a"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "4e453abb8e4dda2c8de2081d1097c0e1"
server_key_iv: "1d20331b99e69b3709ddd076cff5debd"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.289634 21392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:50:24.290357 21418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:24.290549 21392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:50:24.290614 21392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/wal
uuid: "0c88ae23cfa7463c9d83448ef6bf687a"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "4e453abb8e4dda2c8de2081d1097c0e1"
server_key_iv: "1d20331b99e69b3709ddd076cff5debd"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.290705 21392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:24.298537 21392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:24.300349 21392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:24.300493 21392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:24.304843 21392 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.62:32829
I20260810 04:50:24.304879 21499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.62:32829 every 8 connection(s)
I20260810 04:50:24.305279 21392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/data/info.pb
I20260810 04:50:24.306351 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:24.306629 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 21392
I20260810 04:50:24.306740 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-0/wal/instance
I20260810 04:50:24.306926 15284 external_mini_cluster.cc:1428] Setting key 646f1091a467f006a7c822373abdeacb
I20260810 04:50:24.307969 21506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.237.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.61@KRBTEST.COM" created.
I20260810 04:50:24.318960 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.308702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:32829 (local address 127.0.0.1:50330)
0810 04:50:24.308921 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.308968 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.309709 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.310010 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.310013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.310388 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.310636 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.310643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.311698 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.311699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.312912 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.312916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.313478 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.313479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.313593 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.314107 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.314118 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.315726 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.317510 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.317516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.317518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.317704 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.317953 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.317954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.317954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.318038 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.318383 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.318388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.318453 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.318539 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.318718 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260810 04:50:24.318964 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.308770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.14.237.62:32829)
0810 04:50:24.308925 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.308934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.309071 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:50:24.309871 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.309873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.309891 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.309959 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.310733 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.311587 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.313001 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.313366 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.313447 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.315825 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.315832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.315833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.315840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.317429 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.317789 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.317792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.317792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.317873 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.318102 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.318103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.318103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.318301 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.318389 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.318586 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.318792 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260810 04:50:24.319772 21506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:24.320190 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.320047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53820)
0810 04:50:24.320101 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.320155 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:50:24.320360 21423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
I20260810 04:50:24.320856 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.320736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53834)
0810 04:50:24.320790 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.320824 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 04:50:24.321009 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:44059: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab.
Entry for principal kudu/127.14.237.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab.
I20260810 04:50:24.335522 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.61:44059
--webserver_interface=127.14.237.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab
--principal=kudu/127.14.237.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:32829,127.14.237.61:44059,127.14.237.60:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:50:24.370242 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } attempt: 1
I20260810 04:50:24.370736 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.370593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53850)
0810 04:50:24.370665 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.370704 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 04:50:24.371078 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.370982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53862)
0810 04:50:24.371028 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.371059 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:50:24.371158 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:44059: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
W20260810 04:50:24.417485 21518 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:50:24.417742 21518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:24.417771 21518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:24.417819 21518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:24.418876 21518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:24.418941 21518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:24.418963 21518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:24.418982 21518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:24.419010 21518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:24.420485 21518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:32829,127.14.237.61:44059,127.14.237.60:41731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.61.keytab
--principal=kudu/127.14.237.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.61:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.237.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:24.420773 21518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:24.421008 21518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:50:24.425940 21518 server_base.cc:1034] running on GCE node
W20260810 04:50:24.425889 21529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:24.425897 21526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:24.425904 21527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:24.426384 21518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:24.426702 21518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:24.427877 21518 hybrid_clock.cc:648] HybridClock initialized: now 1786337424427867 us; error 36 us; skew 500 ppm
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:24.430481 21518 init.cc:377] Logged in from keytab as kudu/127.14.237.61@KRBTEST.COM (short username kudu)
I20260810 04:50:24.430972 21518 webserver.cc:460] Webserver started at http://127.14.237.61:42479/ using document root <none> and password file <none>
I20260810 04:50:24.431227 21518 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:24.431298 21518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:24.431448 21518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:24.432469 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/data/instance:
uuid: "b7ef7607f1d546b2a6470a5922289348"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "f875a77f75e380643f7d634f167c3cc4"
server_key_iv: "82503cb2fbaa40b26b4264a4c0dad474"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.432840 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/wal/instance:
uuid: "b7ef7607f1d546b2a6470a5922289348"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "f875a77f75e380643f7d634f167c3cc4"
server_key_iv: "82503cb2fbaa40b26b4264a4c0dad474"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.434275 21518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:50:24.435040 21545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:24.435212 21518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:24.435318 21518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/wal
uuid: "b7ef7607f1d546b2a6470a5922289348"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "f875a77f75e380643f7d634f167c3cc4"
server_key_iv: "82503cb2fbaa40b26b4264a4c0dad474"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.435393 21518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:24.439287 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } attempt: 2
I20260810 04:50:24.439649 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.439528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53876)
0810 04:50:24.439585 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.439622 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:50:24.439976 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.439882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53888)
0810 04:50:24.439925 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.439949 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:50:24.440056 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.61:44059: Network error: Client connection negotiation failed: client connection to 127.14.237.61:44059: connect: Connection refused (error 111)
I20260810 04:50:24.450616 21518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:24.452895 21518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:24.453063 21518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:24.457041 21518 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.61:44059
I20260810 04:50:24.457087 21642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.61:44059 every 8 connection(s)
I20260810 04:50:24.457464 21518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/data/info.pb
I20260810 04:50:24.458433 21643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:24.459956 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:24.460983 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 21518
I20260810 04:50:24.461071 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-1/wal/instance
I20260810 04:50:24.461295 15284 external_mini_cluster.cc:1428] Setting key d25f8d555fc9aa4e155749653c5616ee
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:50:24.469642 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.460642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.14.237.62:32829)
0810 04:50:24.460732 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.460734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.460918 (+ 184us) server_negotiation.cc:407] Connection header received
0810 04:50:24.461642 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.461643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.461660 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.461746 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.462418 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.462911 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.463924 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.464154 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.464248 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.466718 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.466720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.466720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.466728 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.468155 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.468592 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.468593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.468593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.468652 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.468931 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.468932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.468932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.469053 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.469127 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.469470 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.469555 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:50:24.469828 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.460567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:32829 (local address 127.0.0.1:50342)
0810 04:50:24.460790 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.460825 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.461512 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.461824 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.461827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.462103 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.462291 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.462296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.463025 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.463027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.463780 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.463783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.464275 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.464276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.464443 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.464926 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.464936 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.466613 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.468274 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.468281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.468283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.468507 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.468725 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.468726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.468726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.468852 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.469118 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.469126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.469337 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.469447 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.469593 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260810 04:50:24.470368 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.237.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.237.60@KRBTEST.COM" created.
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:50:24.477718 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.470669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53900)
0810 04:50:24.470734 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.470743 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.470784 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.471072 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.471073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.471232 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.471282 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.471284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.472121 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.472122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.472604 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.472606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.473172 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.473173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.473320 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.473696 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.473701 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.474847 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.476627 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.476628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.476628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.476824 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.477068 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.477068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.477069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.477120 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.477487 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.477488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.477521 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.477591 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.477660 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 04:50:24.477902 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.470752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.14.237.61:44059)
0810 04:50:24.470903 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.470908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.470929 (+ 21us) server_negotiation.cc:407] Connection header received
0810 04:50:24.470948 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.470950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.470968 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.471022 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.471360 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.472014 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.472678 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.472968 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.473052 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.474935 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.474943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.474943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.474949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.476545 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.476917 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.476921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.476921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.476999 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.477183 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.477184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.477185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.477407 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.477503 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.477664 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.477752 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260810 04:50:24.478530 21643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:24.478888 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.478758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42528)
0810 04:50:24.478807 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.478861 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 04:50:24.479038 21552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
I20260810 04:50:24.479514 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.479403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42534)
0810 04:50:24.479454 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.479485 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:50:24.479669 21643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:41731: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.237.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab.
Entry for principal kudu/127.14.237.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab.
I20260810 04:50:24.491510 15284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwUvI9j/build/release/bin/kudu
/tmp/dist-test-taskwUvI9j/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.60:41731
--webserver_interface=127.14.237.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab
--principal=kudu/127.14.237.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.237.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.237.62:32829,127.14.237.61:44059,127.14.237.60:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:50:24.528916 21643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } attempt: 1
I20260810 04:50:24.529559 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.529388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42536)
0810 04:50:24.529455 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.529505 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 04:50:24.529953 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.529861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42548)
0810 04:50:24.529904 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.529932 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:50:24.530076 21643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:41731: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
I20260810 04:50:24.531190 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } attempt: 3
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:50:24.538609 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.531474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53906)
0810 04:50:24.531520 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.531530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.531606 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.531770 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.531772 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.531980 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.532046 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.532048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.532702 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.532703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.533297 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.533300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.533677 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.533678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.533759 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.534295 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.534300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.535808 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.537460 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.537461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.537462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.537677 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.537900 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.537900 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.537901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.537965 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.538264 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.538265 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.538311 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.538389 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.538494 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 04:50:24.538609 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.531537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.14.237.61:44059)
0810 04:50:24.531595 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.531598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.531609 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:50:24.531671 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.531672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.531683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.531763 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.532130 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.532617 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.533379 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.533602 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.533678 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.535901 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.535902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.535903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.535910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.537365 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.537764 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.537765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.537765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.537827 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.538031 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.538032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.538032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.538182 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.538260 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.538426 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.538497 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:50:24.539044 21506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:24.539433 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.539334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42550)
0810 04:50:24.539383 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.539409 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:50:24.539812 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.539729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42558)
0810 04:50:24.539775 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.539798 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 04:50:24.539911 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:41731: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
I20260810 04:50:24.571128 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } attempt: 1
I20260810 04:50:24.571587 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.571457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42572)
0810 04:50:24.571519 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.571559 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:50:24.571972 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.571872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42582)
0810 04:50:24.571921 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.571951 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:50:24.572062 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:41731: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
W20260810 04:50:24.576057 21653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:50:24.576292 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:50:24.576352 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:50:24.576388 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:50:24.577620 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:50:24.577682 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:50:24.577706 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:50:24.577733 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:50:24.577757 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:50:24.579267 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.237.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.237.62:32829,127.14.237.61:44059,127.14.237.60:41731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/krb5kdc/kudu_127.14.237.60.keytab
--principal=kudu/127.14.237.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.237.60:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.237.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 03:54:01 UTC on ccycloud.quasar-icijmt.root.comops.site
build id 4577009
I20260810 04:50:24.579556 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:50:24.579784 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:50:24.584478 21659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:24.584451 21663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:50:24.584465 21661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:50:24.584627 21653 server_base.cc:1034] running on GCE node
I20260810 04:50:24.584884 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:50:24.585263 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:50:24.586427 21653 hybrid_clock.cc:648] HybridClock initialized: now 1786337424586400 us; error 42 us; skew 500 ppm
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:50:24.588913 21653 init.cc:377] Logged in from keytab as kudu/127.14.237.60@KRBTEST.COM (short username kudu)
I20260810 04:50:24.589459 21653 webserver.cc:460] Webserver started at http://127.14.237.60:35443/ using document root <none> and password file <none>
I20260810 04:50:24.589682 21653 fs_manager.cc:359] Metadata directory not provided
I20260810 04:50:24.589749 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:50:24.589879 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:50:24.590911 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/data/instance:
uuid: "1f4175a89f0440cba412f4ee0173c993"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "8239ab7d8427805dc0a90467963bc29c"
server_key_iv: "50441a9221e519c2d8f3ed85db9f6a01"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.591307 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/wal/instance:
uuid: "1f4175a89f0440cba412f4ee0173c993"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "8239ab7d8427805dc0a90467963bc29c"
server_key_iv: "50441a9221e519c2d8f3ed85db9f6a01"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.592698 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:50:24.593525 21670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:50:24.593674 21653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:50:24.593784 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/data,/tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/wal
uuid: "1f4175a89f0440cba412f4ee0173c993"
format_stamp: "Formatted at 2026-08-10 04:50:24 on dist-test-slave-8hhm"
server_key: "8239ab7d8427805dc0a90467963bc29c"
server_key_iv: "50441a9221e519c2d8f3ed85db9f6a01"
server_key_version: "encryptionkey@0"
I20260810 04:50:24.593858 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:50:24.598260 21643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } attempt: 2
I20260810 04:50:24.598629 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.598512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42590)
0810 04:50:24.598567 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.598603 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:50:24.598958 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.598865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42604)
0810 04:50:24.598908 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.598934 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 04:50:24.599038 21643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.237.60:41731: Network error: Client connection negotiation failed: client connection to 127.14.237.60:41731: connect: Connection refused (error 111)
I20260810 04:50:24.604431 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:50:24.606745 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:50:24.606907 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:50:24.610986 21653 rpc_server.cc:307] RPC server started. Bound to: 127.14.237.60:41731
I20260810 04:50:24.611016 21759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.237.60:41731 every 8 connection(s)
I20260810 04:50:24.611371 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/data/info.pb
I20260810 04:50:24.612353 21760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:50:24.613905 21760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:24.617008 15284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwUvI9j/build/release/bin/kudu as pid 21653
I20260810 04:50:24.617094 15284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwUvI9j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786337145367972-15284-0/minicluster-data/master-2/wal/instance
I20260810 04:50:24.617367 15284 external_mini_cluster.cc:1428] Setting key a8138157ae0daa77ea832e4dbc11e8b6
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
I20260810 04:50:24.622764 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.614570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.14.237.62:32829)
0810 04:50:24.614652 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.614655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.614813 (+ 158us) server_negotiation.cc:407] Connection header received
0810 04:50:24.615257 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.615258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.615268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.615321 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.615869 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.616331 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.617381 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.617630 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.617726 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.619905 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.619906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.619907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.619913 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.621407 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.621804 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.621805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.621806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.621867 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.622119 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.622120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.622120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.622239 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.622330 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.622595 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.622676 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
I20260810 04:50:24.622943 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.614514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.62:32829 (local address 127.0.0.1:50358)
0810 04:50:24.614697 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.614732 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.615173 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.615375 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.615378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.615610 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.615801 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.615805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.616413 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.616414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.617302 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.617305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.617761 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.617763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.617906 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.618449 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.618456 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.619818 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.621493 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.621500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.621501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.621726 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.621938 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.621939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.621939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.622050 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.622323 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.622330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.622464 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.622571 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.622708 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260810 04:50:24.623560 21760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:50:24.626432 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.617741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.14.237.60:41731)
0810 04:50:24.617985 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.617990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.618009 (+ 19us) server_negotiation.cc:407] Connection header received
0810 04:50:24.618057 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.618060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.618087 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.618171 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.618767 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.619432 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.620350 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.620856 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.620951 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.623350 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.623357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.623357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.623364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.625208 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.625575 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.625578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.625578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.625642 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.625869 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.625871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.625871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.626016 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.626201 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.626245 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.626322 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":115,"threads_started":1}
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:50:24.630539 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.623834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.61:44059 (local address 127.0.0.1:53914)
0810 04:50:24.623862 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.623872 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.623905 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.624073 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.624074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.624258 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.624313 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.624315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.625011 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.625012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.625696 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.625698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.626063 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.626064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.626130 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.626548 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.626553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.627729 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.629544 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.629545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.629545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.629768 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.629977 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.629978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.629978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.630047 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.630334 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.630335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.630369 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.630421 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.630485 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260810 04:50:24.630641 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.623898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.14.237.61:44059)
0810 04:50:24.623949 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.623951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.623968 (+ 17us) server_negotiation.cc:407] Connection header received
0810 04:50:24.623991 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.623992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.624000 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.624047 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.624395 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.624930 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.625778 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.625993 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.626066 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.627820 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.627821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.627821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.627829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.629481 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.629848 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.629849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.629849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.629907 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.630120 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.630121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.630121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.630263 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.630338 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.630450 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.630531 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:50:24.631000 21760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.62@KRBTEST.COM
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.60@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
I20260810 04:50:24.635218 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.627275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.14.237.62:32829)
0810 04:50:24.627342 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.627344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.627352 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:50:24.627374 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.627375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.627383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.627433 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.627956 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.628386 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.629399 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.629618 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.629689 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.632327 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.632328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.632329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.632336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.634014 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.634390 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.634391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.634391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.634443 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.634649 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.634650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.634650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.634745 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.634879 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.634966 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.635062 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:50:24.638059 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.631283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42618)
0810 04:50:24.631324 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.631334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.631369 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.631477 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.631478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.631617 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.631650 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.631652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.632532 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.632533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.632969 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.632970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.633481 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.633481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.633523 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.633910 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.633913 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.635259 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.637009 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.637010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.637011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.637178 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.637421 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.637422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.637423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.637480 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.637727 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.637727 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.637765 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.637832 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.637935 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 04:50:24.638123 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.631339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.14.237.60:41731)
0810 04:50:24.631376 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.631378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.631388 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:50:24.631409 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.631410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.631420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.631473 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.631723 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.632232 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.633051 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.633397 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.633473 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.635359 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.635361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.635361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.635368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.636904 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.637292 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.637293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.637294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.637345 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.637552 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.637553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.637553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.637658 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.637734 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.637908 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.638010 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.237.61@KRBTEST.COM
I20260810 04:50:24.639778 21760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993: Bootstrap starting.
I20260810 04:50:24.640623 21760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:24.640915 21760 log.cc:824] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:24.641803 21760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993: No bootstrap required, opened a new log
I20260810 04:50:24.643016 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.635830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.14.237.61:44059)
0810 04:50:24.635872 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.635873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.635883 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:50:24.635931 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.635932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.635941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.636011 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.636429 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.636866 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.637588 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.637793 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.637869 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.640190 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.640191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.640191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.640198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.641837 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.642209 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.642210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.642210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.642267 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.642492 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.642493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.642493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.642605 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.642790 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.642828 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.642903 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:50:24.643476 21760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } }
I20260810 04:50:24.643666 21760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4175a89f0440cba412f4ee0173c993, State: Initialized, Role: FOLLOWER
I20260810 04:50:24.643746 21760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } }
I20260810 04:50:24.644147 21760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:24.644130 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.644241 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:24.644733 15284 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 04:50:24.646206 21790 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:50:24.646265 21790 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:50:24.646025 21462 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0c88ae23cfa7463c9d83448ef6bf687a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50372
I20260810 04:50:24.647198 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } attempt: 2
I20260810 04:50:24.646507 21653 master_runner.cc:186] Error getting master registration for 127.14.237.62:32829: OK, instance_id { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" instance_seqno: 1786337424303676 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:50:24.648787 21653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.62@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
I20260810 04:50:24.655543 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.647672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.14.237.60:41731)
0810 04:50:24.647756 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.647758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.647768 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:50:24.647788 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.647789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.647799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.647860 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.648189 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.648857 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.650146 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.650521 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.650604 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.652588 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.652589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.652589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.652596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.654172 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.654626 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.654627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.654627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.654695 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.654939 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.654939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.654940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.655096 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.655162 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.655359 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.655436 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:50:24.655546 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.647423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42620)
0810 04:50:24.647504 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.647514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.647550 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.647876 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.647877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.648051 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.648098 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.648100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.648971 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.648972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.649999 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.650003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.650610 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.650611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.650670 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.651135 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.651140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.652479 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.654273 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.654275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.654275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.654545 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.654780 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.654781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.654781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.654864 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.655177 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.655178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.655235 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.655324 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.655430 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 04:50:24.657341 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a: Bootstrap starting.
I20260810 04:50:24.658182 21506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:24.658589 21506 log.cc:824] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:24.659523 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a: No bootstrap required, opened a new log
I20260810 04:50:24.661020 21506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } }
I20260810 04:50:24.661191 21506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c88ae23cfa7463c9d83448ef6bf687a, State: Initialized, Role: FOLLOWER
I20260810 04:50:24.661342 21506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } }
I20260810 04:50:24.661677 21506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:24.661659 21795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.661723 21462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 04:50:24.661839 21795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:24.661878 21462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:24.662699 21462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } }
I20260810 04:50:24.662961 21462 leader_election.cc:288] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [CANDIDATE]: Term 1 election: Requested vote from peers b7ef7607f1d546b2a6470a5922289348 (127.14.237.61:44059), 1f4175a89f0440cba412f4ee0173c993 (127.14.237.60:41731)
I20260810 04:50:24.663563 21717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f4175a89f0440cba412f4ee0173c993"
I20260810 04:50:24.663708 21717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:24.664626 21717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c88ae23cfa7463c9d83448ef6bf687a in term 1.
I20260810 04:50:24.664980 21428 leader_election.cc:302] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c88ae23cfa7463c9d83448ef6bf687a, 1f4175a89f0440cba412f4ee0173c993; no voters:
I20260810 04:50:24.665138 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:50:24.665303 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [term 1 LEADER]: Becoming Leader. State: Replica: 0c88ae23cfa7463c9d83448ef6bf687a, State: Running, Role: LEADER
I20260810 04:50:24.665401 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } }
I20260810 04:50:24.665874 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c88ae23cfa7463c9d83448ef6bf687a. Latest consensus state: current_term: 1 leader_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.666033 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:24.667409 21392 master_runner.cc:186] Error getting master registration for 127.14.237.61:44059: OK, instance_id { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" instance_seqno: 1786337424455881 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:50:24.669885 21392 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:50:24.666402 21834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:50:24.670516 21834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:50:24.672168 21717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [term 1 FOLLOWER]: Refusing update from remote peer 0c88ae23cfa7463c9d83448ef6bf687a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:50:24.672554 21821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:50:24.672991 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.665119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.14.237.61:44059)
0810 04:50:24.665161 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.665163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.665178 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:50:24.665202 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.665203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.665232 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.665287 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.666046 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.666761 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.667742 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.668014 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.668056 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.669630 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.669631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.669632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.669639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.671333 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.671732 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.671733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.671733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.671814 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.672066 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.672067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.672067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.672207 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.672283 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.672783 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.672884 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:50:24.674791 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c88ae23cfa7463c9d83448ef6bf687a. Latest consensus state: current_term: 1 leader_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.674887 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:24.678171 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.664067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.14.237.62:32829)
0810 04:50:24.664485 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.664487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.667566 (+ 3079us) server_negotiation.cc:407] Connection header received
0810 04:50:24.667720 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.667721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.667744 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.667776 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.669021 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.669810 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.671569 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.671931 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.671979 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.674418 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.674419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.674420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.674427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.676257 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.676972 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.676974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.676974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.677045 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.677400 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.677401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.677401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.677548 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.677624 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.677979 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.678063 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260810 04:50:24.679164 21795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.679267 21795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:24.679333 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.679400 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:24.679702 21837 mvcc.cc:205] Tried to move back new op lower bound from 7316838091454451712 to 7316838091430125568. Current Snapshot: MvccSnapshot[applied={T|T < 7316838091454451712}]
I20260810 04:50:24.685288 21842 mvcc.cc:205] Tried to move back new op lower bound from 7316838091454451712 to 7316838091430125568. Current Snapshot: MvccSnapshot[applied={T|T < 7316838091454451712}]
I20260810 04:50:24.689107 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.664163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.14.237.60:41731)
0810 04:50:24.664454 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.664455 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.672732 (+ 8277us) server_negotiation.cc:407] Connection header received
0810 04:50:24.674882 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.674883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.674894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.674926 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.679934 (+ 5008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.680764 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.682481 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.682803 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.682843 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.685346 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.685348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.685348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.685356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.687139 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.687798 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.687800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.687800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.687864 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.688221 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.688222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.688223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.688363 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.688491 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.688906 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.689003 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260810 04:50:24.690155 21643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } attempt: 3
I20260810 04:50:24.690913 21834 catalog_manager.cc:1347] Generated new cluster ID: dfc9eb7441be4b35a97fb249a46e91c9
I20260810 04:50:24.690965 21834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:50:24 dist-test-slave-8hhm krb5kdc[21368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786337424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.237.61@KRBTEST.COM for kudu/127.14.237.60@KRBTEST.COM
I20260810 04:50:24.698199 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.690412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.237.60:41731 (local address 127.0.0.1:42646)
0810 04:50:24.690458 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:50:24.690468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:50:24.690503 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:50:24.690670 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:50:24.690671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:50:24.690843 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:50:24.690890 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.690891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.691910 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.691912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.692854 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:50:24.692857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.693534 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.693535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:50:24.693610 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.693997 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:50:24.694002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:50:24.695487 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:50:24.697250 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.697250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.697251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.697444 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.697623 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:50:24.697623 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:50:24.697624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:50:24.697670 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:50:24.697895 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:50:24.697896 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:50:24.697936 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 04:50:24.697992 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:50:24.698080 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 04:50:24.698496 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:24.690462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.14.237.60:41731)
0810 04:50:24.690510 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.690512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.690523 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:50:24.690556 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.690556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.690565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.690621 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.690992 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.691774 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.693017 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:24.693387 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:24.693465 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:24.695589 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:50:24.695590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:50:24.695590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:50:24.695596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:50:24.697132 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.697515 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.697516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.697517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.697565 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:50:24.697726 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:50:24.697726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:50:24.697726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:50:24.697828 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:50:24.697875 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:24.698328 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 04:50:24.698416 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:50:24.699950 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348: Bootstrap starting.
I20260810 04:50:24.700711 21643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348: Neither blocks nor log segments found. Creating new log.
I20260810 04:50:24.701004 21643 log.cc:824] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348: Log is configured to *not* fsync() on all Append() calls
I20260810 04:50:24.701862 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348: No bootstrap required, opened a new log
I20260810 04:50:24.703825 21643 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } }
I20260810 04:50:24.703965 21643 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ef7607f1d546b2a6470a5922289348, State: Initialized, Role: FOLLOWER
I20260810 04:50:24.704043 21643 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } }
I20260810 04:50:24.704348 21643 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:50:24.704396 21594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7ef7607f1d546b2a6470a5922289348"
I20260810 04:50:24.704458 21593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:50:24.705478 21593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [term 1 FOLLOWER]: Refusing update from remote peer 0c88ae23cfa7463c9d83448ef6bf687a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:50:24.706462 21594 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c88ae23cfa7463c9d83448ef6bf687a in term 1.
I20260810 04:50:24.707006 21821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:50:24.710039 21871 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.710137 21871 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:24.710186 21871 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c88ae23cfa7463c9d83448ef6bf687a. Latest consensus state: current_term: 1 leader_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.710233 21871 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:24.710268 21871 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.710317 21871 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:50:24.710484 21872 mvcc.cc:205] Tried to move back new op lower bound from 7316838091454451712 to 7316838091430125568. Current Snapshot: MvccSnapshot[applied={T|T < 7316838091454451712}]
I20260810 04:50:24.713841 21868 catalog_manager.cc:1259] Loaded cluster ID: dfc9eb7441be4b35a97fb249a46e91c9
I20260810 04:50:24.713896 21868 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348: loading cluster ID for follower catalog manager: success
W20260810 04:50:24.713989 21868 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260810 04:50:24.714041 21868 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:50:24.714483 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88ae23cfa7463c9d83448ef6bf687a" member_type: VOTER last_known_addr { host: "127.14.237.62" port: 32829 } } peers { permanent_uuid: "b7ef7607f1d546b2a6470a5922289348" member_type: VOTER last_known_addr { host: "127.14.237.61" port: 44059 } } peers { permanent_uuid: "1f4175a89f0440cba412f4ee0173c993" member_type: VOTER last_known_addr { host: "127.14.237.60" port: 41731 } } }
I20260810 04:50:24.714605 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a [sys.catalog]: This master's current role is: LEADER
I20260810 04:50:24.716305 21834 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:50:24.716810 21834 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:50:24.726495 21834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c88ae23cfa7463c9d83448ef6bf687a: Generated new TSK 0
I20260810 04:50:24.726678 21834 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:50:24.754714 21515 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.752808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.14.237.62:32829)
0810 04:50:24.752873 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.752875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.753945 (+ 1070us) server_negotiation.cc:407] Connection header received
0810 04:50:24.754059 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.754060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.754072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.754105 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.754638 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50384: BlockingRecv error: recv got EOF from 127.0.0.1:50384 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260810 04:50:24.755512 21766 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.753198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.14.237.60:41731)
0810 04:50:24.753255 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.753257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.754981 (+ 1724us) server_negotiation.cc:407] Connection header received
0810 04:50:24.755087 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.755088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.755099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.755132 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.755452 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42648: BlockingRecv error: recv got EOF from 127.0.0.1:42648 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 04:50:24.756326 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.753609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.14.237.61:44059)
0810 04:50:24.753652 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.753654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.755765 (+ 2111us) server_negotiation.cc:407] Connection header received
0810 04:50:24.755891 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.755892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.755904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.755936 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.756261 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53946: BlockingRecv error: recv got EOF from 127.0.0.1:53946 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:50:24.758894 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.758086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.14.237.61:44059)
0810 04:50:24.758130 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.758131 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.758346 (+ 215us) server_negotiation.cc:407] Connection header received
0810 04:50:24.758468 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.758469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.758487 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.758518 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.758847 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53950: BlockingRecv error: recv got EOF from 127.0.0.1:53950 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 04:50:24.762179 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.761302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.14.237.61:44059)
0810 04:50:24.761339 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.761341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.761650 (+ 309us) server_negotiation.cc:407] Connection header received
0810 04:50:24.761759 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.761760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.761770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.761801 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.762135 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53966: BlockingRecv error: recv got EOF from 127.0.0.1:53966 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 04:50:24.766189 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.765373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.14.237.61:44059)
0810 04:50:24.765408 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.765410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.765662 (+ 252us) server_negotiation.cc:407] Connection header received
0810 04:50:24.765762 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.765763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.765775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.765806 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.766143 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53972: BlockingRecv error: recv got EOF from 127.0.0.1:53972 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 04:50:24.772253 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.771407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.14.237.61:44059)
0810 04:50:24.771444 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.771446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.771699 (+ 253us) server_negotiation.cc:407] Connection header received
0810 04:50:24.771819 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.771819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.771831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.771863 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.772205 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53976: BlockingRecv error: recv got EOF from 127.0.0.1:53976 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:50:24.782254 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.781481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.14.237.61:44059)
0810 04:50:24.781514 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.781516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.781740 (+ 224us) server_negotiation.cc:407] Connection header received
0810 04:50:24.781833 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.781834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.781844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.781872 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.782205 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53988: BlockingRecv error: recv got EOF from 127.0.0.1:53988 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:50:24.801450 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.800447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.14.237.61:44059)
0810 04:50:24.800489 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.800491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.800777 (+ 286us) server_negotiation.cc:407] Connection header received
0810 04:50:24.800880 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.800881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.800892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.800921 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.801392 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54004: BlockingRecv error: recv got EOF from 127.0.0.1:54004 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 04:50:24.836086 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.835043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.14.237.61:44059)
0810 04:50:24.835090 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.835093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.835416 (+ 323us) server_negotiation.cc:407] Connection header received
0810 04:50:24.835530 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.835532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.835546 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.835578 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.836016 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54014: BlockingRecv error: recv got EOF from 127.0.0.1:54014 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:50:24.902407 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:24.901639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.14.237.61:44059)
0810 04:50:24.901714 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:24.901717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:24.901837 (+ 120us) server_negotiation.cc:407] Connection header received
0810 04:50:24.901932 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:24.901933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:24.901947 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:24.901994 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:24.902348 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54018: BlockingRecv error: recv got EOF from 127.0.0.1:54018 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 04:50:25.032565 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:25.031806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.14.237.61:44059)
0810 04:50:25.031873 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:25.031879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:25.031975 (+ 96us) server_negotiation.cc:407] Connection header received
0810 04:50:25.032022 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:25.032023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:25.032037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:25.032116 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:25.032471 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54032: BlockingRecv error: recv got EOF from 127.0.0.1:54032 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 04:50:25.290398 21648 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:50:25.289822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.14.237.61:44059)
0810 04:50:25.289892 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:25.289894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:25.289904 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:50:25.289932 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:25.289933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:25.289955 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:25.290024 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:50:25.290347 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54046: BlockingRecv error: recv got EOF from 127.0.0.1:54046 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:50:25.646694 21790 catalog_manager.cc:1259] Loaded cluster ID: dfc9eb7441be4b35a97fb249a46e91c9
I20260810 04:50:25.646777 21790 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993: loading cluster ID for follower catalog manager: success
I20260810 04:50:25.647578 21790 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993: acquiring CA information for follower catalog manager: success
I20260810 04:50:25.647887 21790 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f4175a89f0440cba412f4ee0173c993: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:50:25.715044 21868 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348: acquiring CA information for follower catalog manager: success
I20260810 04:50:25.715412 21868 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7ef7607f1d546b2a6470a5922289348: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:50:25.806767 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:25.803668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.14.237.61:44059)
0810 04:50:25.803820 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:25.803822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:25.803834 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:50:25.803919 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:25.803920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:25.804054 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:25.804140 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:25.804323 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.804957 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.805874 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.806117 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.806195 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:25.806390 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:25.806509 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:25.806591 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:25.806641 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:50:25.806661 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260810 04:50:25.810765 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:25.808558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.14.237.60:41731)
0810 04:50:25.808642 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:25.808643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:25.808686 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:50:25.808772 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:25.808773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:25.808806 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:25.808872 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:25.808941 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.809466 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.810114 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.810295 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.810346 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:25.810469 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:25.810554 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:25.810609 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:25.810640 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 04:50:25.810656 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":43,"threads_started":1}
I20260810 04:50:25.815225 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:25.812615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.14.237.60:41731)
0810 04:50:25.812667 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:25.812669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:25.812676 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:50:25.812740 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:25.812741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:25.812752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:25.812803 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:25.812949 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.813477 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.814467 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.814744 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.814816 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:25.814971 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:25.815001 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:25.815056 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:25.815128 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 04:50:25.815152 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:50:25.815551 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:25.812433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.14.237.61:44059)
0810 04:50:25.812490 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:25.812492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:25.812673 (+ 181us) server_negotiation.cc:407] Connection header received
0810 04:50:25.812794 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:25.812795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:25.812805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:25.812879 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:25.812996 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.813706 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.814631 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.814896 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.814967 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:25.815260 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:25.815294 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:25.815354 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:25.815459 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 04:50:25.815481 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:50:25.816002 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:50:25.812387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.14.237.62:32829)
0810 04:50:25.812549 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 04:50:25.812551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:50:25.812646 (+ 95us) server_negotiation.cc:407] Connection header received
0810 04:50:25.812806 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:50:25.812807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:50:25.812859 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:50:25.812937 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:50:25.813157 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.813961 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.814859 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:50:25.815194 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:50:25.815255 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:50:25.815473 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:50:25.815587 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:50:25.815772 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 04:50:25.815891 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 04:50:25.815919 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":56,"threads_started":1}
I20260810 04:50:25.817855 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 21392
I20260810 04:50:25.834408 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 21518
I20260810 04:50:25.846448 15284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwUvI9j/build/release/bin/kudu with pid 21653
2026-08-10T04:50:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3066 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6132 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280492 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST